Nov 03 19:58:07.845843 ip-10-0-143-81 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 19:58:07.845858 ip-10-0-143-81 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 19:58:07.845867 ip-10-0-143-81 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 19:58:07.846193 ip-10-0-143-81 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 19:58:18.048857 ip-10-0-143-81 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 19:58:18.048875 ip-10-0-143-81 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ec7131f169814c53b926086962e49496 -- Nov 03 20:00:16.877770 ip-10-0-143-81 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:00:17.334862 ip-10-0-143-81 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:00:17.334862 ip-10-0-143-81 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:00:17.334862 ip-10-0-143-81 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:00:17.335325 ip-10-0-143-81 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:00:17.338242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338175 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.338242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338242 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338246 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338249 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338252 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338256 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338258 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338261 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338264 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338267 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338269 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338272 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338274 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338277 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338281 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338283 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338286 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338288 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338291 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338293 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338296 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.338309 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338299 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338301 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338304 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338306 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338309 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338312 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338315 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338318 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338321 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338324 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338326 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338328 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338331 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338333 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338336 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338338 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338341 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338344 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338347 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.338804 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338349 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338352 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338354 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338357 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338360 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338362 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338364 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338367 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338369 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338372 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338374 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338377 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338379 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338382 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338384 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338388 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338390 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338393 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338396 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.339253 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338398 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338401 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338403 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338406 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338408 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338412 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338417 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338420 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338423 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338426 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338428 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338430 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338433 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338436 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338438 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338441 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338444 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338446 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338449 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338451 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.339736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338454 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338457 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338460 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338462 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338465 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338467 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338470 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338472 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338476 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338479 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338481 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338484 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338486 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338489 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338493 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338497 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338500 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338517 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338520 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.340258 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338522 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338525 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338528 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338530 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338533 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338535 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338542 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.338546 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339372 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339379 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339382 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339385 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339388 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339391 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339393 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339396 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339398 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339401 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339404 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339407 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.341041 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339410 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339413 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339415 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339418 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339421 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339423 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339426 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339428 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339431 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339434 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339436 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339439 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339442 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339444 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339447 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339449 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339452 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339454 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339457 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339465 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.341811 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339468 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339470 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339473 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339475 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339478 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339480 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339482 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339485 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339487 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339490 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339493 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339496 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339498 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339501 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339519 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339522 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339524 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339527 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339529 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.342302 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339532 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339534 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339537 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339539 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339542 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339544 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339547 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339549 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339552 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339554 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339557 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339560 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339563 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339570 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339573 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339576 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339578 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339582 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339586 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.342784 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339588 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339591 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339595 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339598 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339600 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339603 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339606 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339609 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339611 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339613 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339616 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339619 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339621 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339624 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339626 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339629 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339631 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339635 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339639 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.343242 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339641 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339644 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339647 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339650 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339654 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339657 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339660 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339662 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339673 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339676 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339678 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339681 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339683 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339685 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339688 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339691 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.339694 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.340981 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.340995 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341000 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.343715 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341004 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341008 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341012 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341016 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341020 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341025 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341029 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341033 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341042 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341046 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341051 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341055 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341059 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341063 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341067 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341071 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341075 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341108 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341112 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.344279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341116 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341149 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341158 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341162 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341166 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341170 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341173 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341178 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341182 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341187 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341191 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341195 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341199 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341203 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341212 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341216 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341220 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341224 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341228 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341231 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.344782 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341235 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341239 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341244 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341248 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341260 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341264 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341269 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341278 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341282 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341286 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341290 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341404 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341595 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341603 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341608 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341619 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341623 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341626 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341628 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341631 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.345256 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341634 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341636 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341639 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341641 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341644 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341646 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341650 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341652 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341654 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341657 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341659 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341662 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341664 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341667 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341671 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341674 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341677 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341680 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341682 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.345797 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341685 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341687 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341690 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341692 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341695 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341698 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341701 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341704 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341707 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341710 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341712 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341715 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341718 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341720 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341723 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341726 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341728 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341731 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341733 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.346273 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341736 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341739 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341741 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341743 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341748 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.341750 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341849 2572 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341856 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341864 2572 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341869 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341873 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341877 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341881 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341885 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341888 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341891 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341895 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341898 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341903 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341906 2572 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341909 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341912 2572 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341915 2572 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341918 2572 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:00:17.346773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341923 2572 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341925 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341928 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341931 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341935 2572 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341939 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341942 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341945 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341948 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341951 2572 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341954 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341957 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341960 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341963 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341967 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341971 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341974 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341977 2572 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341980 2572 flags.go:64] FLAG: --enable-server="true" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341983 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341987 2572 flags.go:64] FLAG: --event-burst="100" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341990 2572 flags.go:64] FLAG: --event-qps="50" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341993 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341996 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:00:17.347336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.341999 2572 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342005 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342009 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342013 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342016 2572 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342018 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342021 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342024 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342027 2572 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342030 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342033 2572 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342036 2572 flags.go:64] FLAG: --feature-gates="" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342040 2572 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342043 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342046 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342049 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342052 2572 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342055 2572 flags.go:64] FLAG: --help="false" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342058 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342061 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342064 2572 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342067 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342070 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:00:17.347974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342073 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342076 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342079 2572 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342081 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342084 2572 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342087 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342090 2572 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342092 2572 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342095 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342098 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342101 2572 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342106 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342109 2572 flags.go:64] FLAG: --lock-file="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342112 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342114 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342117 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342123 2572 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342125 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342128 2572 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342131 2572 flags.go:64] FLAG: --logging-format="text" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342134 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342137 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342140 2572 flags.go:64] FLAG: --manifest-url="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342143 2572 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342147 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:00:17.348544 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342150 2572 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342154 2572 flags.go:64] FLAG: --max-pods="110" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342157 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342160 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342162 2572 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342165 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342168 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342171 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342174 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342181 2572 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342184 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342187 2572 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342190 2572 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342193 2572 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342196 2572 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342199 2572 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342202 2572 flags.go:64] FLAG: --port="10250" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342205 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342210 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-012a04d55116578a2" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342214 2572 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342217 2572 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342221 2572 flags.go:64] FLAG: --register-node="true" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342223 2572 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342231 2572 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342234 2572 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342237 2572 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:00:17.349137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342240 2572 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342244 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342247 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342249 2572 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342252 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342255 2572 flags.go:64] FLAG: --runonce="false" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342258 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342261 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342264 2572 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342266 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342269 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342272 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342275 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342278 2572 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342281 2572 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342283 2572 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342286 2572 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342289 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342292 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342295 2572 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342297 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342304 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342307 2572 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342310 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342317 2572 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:00:17.349820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342320 2572 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342323 2572 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342326 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342329 2572 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342332 2572 flags.go:64] FLAG: --v="2" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342336 2572 flags.go:64] FLAG: --version="false" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342340 2572 flags.go:64] FLAG: --vmodule="" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342343 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342347 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342487 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342491 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342494 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342497 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342499 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342502 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342525 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342529 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342531 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342534 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342537 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342539 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.350407 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342542 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342544 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342547 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342550 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342552 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342555 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342557 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342560 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342562 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342565 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342570 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342572 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342575 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342578 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342581 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342584 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342586 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342589 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342592 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342594 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.350997 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342597 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342599 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342602 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342605 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342607 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342609 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342612 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342614 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342617 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342619 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342622 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342624 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342627 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342629 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342632 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342635 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342639 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342642 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342644 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.351463 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342646 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342649 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342651 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342656 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342659 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342661 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342664 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342668 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342670 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342673 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342675 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342678 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342681 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342683 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342686 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342688 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342690 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342693 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342696 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342699 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.351937 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342701 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342704 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342707 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342709 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342712 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342714 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342717 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342719 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342722 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342725 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342727 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342729 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342732 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342734 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342737 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342741 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342744 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342748 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342752 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.352403 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342755 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342758 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342761 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342763 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342766 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342768 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342771 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342774 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342776 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342779 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342781 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342784 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342787 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342790 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342792 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342795 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.352871 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.342823 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342860 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342864 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342866 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342869 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342872 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342875 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342877 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342880 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342882 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342885 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342887 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342891 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342895 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342897 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342900 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342902 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342905 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342909 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.353247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342913 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342916 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342918 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342920 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342923 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342925 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342928 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342930 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342933 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342936 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342938 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342941 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342943 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342945 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342948 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342950 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342953 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342955 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342958 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.353727 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342960 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342963 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342965 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342968 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342970 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342972 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342977 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342981 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342983 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342986 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342989 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342992 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342995 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.342997 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343000 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343002 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343005 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343007 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343010 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343012 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.354170 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343015 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343018 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343022 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343024 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343028 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343031 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343034 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343036 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343039 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343042 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343044 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343046 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343049 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343052 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343054 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343056 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343059 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343061 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343065 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.354718 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343069 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343072 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343074 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343077 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343079 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343082 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343084 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343087 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343089 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343092 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343094 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343097 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343099 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343101 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343104 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343107 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343109 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343112 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343114 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343116 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.355194 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343119 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343121 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343124 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343126 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343128 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343131 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343133 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343136 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343138 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.343140 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.349591 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.349605 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.349624 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.349629 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349686 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349691 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349694 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349698 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349701 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.355676 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349704 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349707 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349709 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349712 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349714 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349717 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349720 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349722 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349725 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349727 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349729 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349732 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349736 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349740 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349743 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349746 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349748 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349751 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349753 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349755 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.356119 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349758 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349761 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349763 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349766 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349769 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349772 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349775 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349777 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349780 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349782 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349785 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349787 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349791 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349794 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349797 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349800 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349803 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349806 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349808 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.356591 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349811 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349813 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349816 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349819 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349821 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349824 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349827 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349829 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349832 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349835 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349838 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349841 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349844 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349846 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349849 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349851 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349854 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349856 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349859 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.357045 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349862 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349865 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349867 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349870 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349872 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349875 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349877 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349879 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349882 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349884 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349887 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349889 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349892 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349894 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349896 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349899 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349901 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349904 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349906 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.357490 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349909 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349911 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349913 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349916 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349918 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349921 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349923 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349926 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349928 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349931 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349933 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349936 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349939 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349942 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349945 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349947 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349950 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349953 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349955 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349958 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.357943 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349960 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349962 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349965 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.349968 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.349997 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350159 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350164 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350167 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350170 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350172 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350175 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350178 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350180 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350198 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350202 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350205 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.358404 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350208 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350210 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350213 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350215 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350218 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350221 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350224 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350226 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350229 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350231 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350244 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350247 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350249 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350252 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350254 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350256 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350259 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350261 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350264 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350266 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.358922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350269 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350271 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350274 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350276 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350278 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350282 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350286 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350290 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350292 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350295 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350297 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350300 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350302 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350305 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350307 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350310 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350312 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350315 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350320 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.359392 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350324 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350327 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350331 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350333 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350337 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350340 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350343 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350346 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350348 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350351 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350353 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350356 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350358 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350361 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350363 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350366 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350369 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350371 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350373 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350376 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.359872 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350378 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350381 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350383 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350386 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350389 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350391 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350394 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350396 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350399 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350401 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350404 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350408 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350411 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350414 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350416 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350419 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350422 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350425 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350427 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350430 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.360337 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350432 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350435 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350437 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350440 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350442 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350445 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350447 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350450 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350452 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350455 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350457 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350460 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350462 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350464 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350467 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350469 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.360819 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.350492 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350558 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350562 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350566 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350569 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350571 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350574 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350577 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350582 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350585 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350589 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350592 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350595 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350598 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350601 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350603 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350606 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350608 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350611 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:17.361195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350613 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350616 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350618 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350620 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350623 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350626 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350628 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350631 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350633 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350636 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350639 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350641 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350644 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350646 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350649 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350651 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350654 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350656 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350659 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:17.361669 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350661 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350664 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350667 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350670 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350673 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350675 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350678 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350680 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350683 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350686 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350688 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350691 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350693 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350695 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350699 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350701 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350704 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350706 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350709 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350711 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:17.362144 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350713 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350716 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350720 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350722 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350725 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350727 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350729 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350732 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350734 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350737 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350739 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350741 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350743 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350746 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350750 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350752 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350755 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350757 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350760 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350762 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:17.362686 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350765 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350767 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350770 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350773 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350776 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350778 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350781 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350783 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350786 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350788 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350790 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350793 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350795 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350798 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350801 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350803 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350806 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350808 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350811 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350813 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:17.363368 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350815 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350818 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350820 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350825 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350828 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350831 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350834 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350837 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:17.350839 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.350889 2572 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.352800 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.353739 2572 server.go:1014] "Starting client certificate rotation" Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.353841 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.354384 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.356276 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:00:17.363856 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.357792 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:00:17.369318 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.369296 2572 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:00:17.372836 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.372823 2572 log.go:25] "Validated CRI v1 image API" Nov 03 20:00:17.373720 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.373698 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:00:17.377901 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.377805 2572 fs.go:135] Filesystem UUIDs: map[12621e8d-2a8b-4cc6-bda6-89c9305ff98e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 87584bb3-24e6-4223-90b4-3b26dce39c69:/dev/nvme0n1p3] Nov 03 20:00:17.377987 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.377904 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:00:17.384306 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.384189 2572 manager.go:217] Machine: {Timestamp:2026-11-03 20:00:17.382821846 +0000 UTC m=+0.379593034 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108365 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bc8b3541204e6f69f5c5c1a9f15a SystemUUID:ec24bc8b-3541-204e-6f69-f5c5c1a9f15a BootID:ec7131f1-6981-4c53-b926-086962e49496 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:22:f9:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:22:f9:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:cc:b2:7e:69:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:00:17.384306 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.384297 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:00:17.384475 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.384391 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:00:17.386851 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.386822 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:00:17.386984 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.386853 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-81.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:00:17.387034 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.386986 2572 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:00:17.387034 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.386995 2572 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:00:17.387034 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.387033 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:00:17.387108 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.387052 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:00:17.387852 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.387842 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:00:17.391296 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.391283 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:00:17.391420 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.391411 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:00:17.392151 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.392142 2572 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:00:17.392189 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.392156 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:00:17.392189 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.392166 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:00:17.392189 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.392175 2572 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:00:17.392189 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.392183 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:00:17.392467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.392455 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:00:17.396253 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.396226 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:00:17.399760 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.399741 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:00:17.400841 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.400828 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:00:17.401537 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401524 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:00:17.401584 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401556 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:00:17.401722 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401713 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:00:17.401757 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401728 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:00:17.401757 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401734 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:00:17.401757 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401740 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:00:17.401757 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401746 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:00:17.401757 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.401752 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:00:17.402325 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.402317 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:00:17.402353 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.402326 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:00:17.402353 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.402333 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:00:17.402353 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.402339 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:00:17.402353 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.402348 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:00:17.402469 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.402357 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:00:17.403175 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.403165 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:00:17.403175 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.403174 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:00:17.407007 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.406991 2572 server.go:1263] "Started kubelet" Nov 03 20:00:17.407202 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.407125 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:00:17.407274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.407196 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:00:17.407321 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.407283 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:00:17.407786 ip-10-0-143-81 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:00:17.408528 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.408380 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:00:17.410478 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.410461 2572 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:00:17.412372 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.412350 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-81.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:00:17.413451 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.412350 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-81.us-east-2.compute.internal.18e4a1677a5ae9e3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-81.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:17.406781923 +0000 UTC m=+0.403553113,LastTimestamp:2026-11-03 20:00:17.406781923 +0000 UTC m=+0.403553113,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-81.us-east-2.compute.internal,}" Nov 03 20:00:17.414760 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.414743 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:00:17.415524 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.415475 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:00:17.415908 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.415895 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:00:17.416050 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.416032 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416565 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416571 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416611 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416671 2572 factory.go:55] Registering systemd factory Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416683 2572 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416687 2572 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:00:17.416807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416700 2572 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:00:17.417089 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416839 2572 factory.go:153] Registering CRI-O factory Nov 03 20:00:17.417089 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416848 2572 factory.go:223] Registration of the crio container factory successfully Nov 03 20:00:17.417089 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.416925 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:00:17.417214 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.417103 2572 factory.go:103] Registering Raw factory Nov 03 20:00:17.417214 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.417123 2572 manager.go:1199] Started watching for new ooms in manager Nov 03 20:00:17.419715 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.419701 2572 manager.go:319] Starting recovery of all containers Nov 03 20:00:17.423472 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.423440 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 20:00:17.423623 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.423597 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-81.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:00:17.423743 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.423721 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-81.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 20:00:17.430948 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.430924 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 20:00:17.431087 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.431043 2572 manager.go:324] Recovery completed Nov 03 20:00:17.434829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.434812 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.436546 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.436529 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.436604 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.436559 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.436604 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.436570 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.437084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437070 2572 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:00:17.437084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437078 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.437181 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437081 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:00:17.437181 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437133 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:00:17.437754 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437739 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.437820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437764 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.437820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.437774 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.439407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.439395 2572 policy_none.go:50] "Start" Nov 03 20:00:17.439454 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.439411 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:00:17.439454 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.439426 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:00:17.441551 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.441539 2572 policy_none.go:44] "Start" Nov 03 20:00:17.442218 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.442136 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c2111ad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-81.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-81.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:17.436545453 +0000 UTC m=+0.433316641,LastTimestamp:2026-11-03 20:00:17.436545453 +0000 UTC m=+0.433316641,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-81.us-east-2.compute.internal,}" Nov 03 20:00:17.449493 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.449429 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c21597b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-81.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-81.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:17.436563835 +0000 UTC m=+0.433335024,LastTimestamp:2026-11-03 20:00:17.436563835 +0000 UTC m=+0.433335024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-81.us-east-2.compute.internal,}" Nov 03 20:00:17.450808 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.450791 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sq6zv" Nov 03 20:00:17.452668 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.452613 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c218061 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-81.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-81.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:17.436573793 +0000 UTC m=+0.433344982,LastTimestamp:2026-11-03 20:00:17.436573793 +0000 UTC m=+0.433344982,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-81.us-east-2.compute.internal,}" Nov 03 20:00:17.458185 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.458083 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c2111ad\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c2111ad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-81.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-81.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:17.436545453 +0000 UTC m=+0.433316641,LastTimestamp:2026-11-03 20:00:17.437751994 +0000 UTC m=+0.434523183,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-81.us-east-2.compute.internal,}" Nov 03 20:00:17.460522 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.460398 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sq6zv" Nov 03 20:00:17.462081 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.462023 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c21597b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-81.us-east-2.compute.internal.18e4a1677c21597b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-81.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-81.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-81.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:17.436563835 +0000 UTC m=+0.433335024,LastTimestamp:2026-11-03 20:00:17.437768445 +0000 UTC m=+0.434539633,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-81.us-east-2.compute.internal,}" Nov 03 20:00:17.473051 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.473022 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:00:17.475370 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475355 2572 manager.go:341] "Starting Device Plugin manager" Nov 03 20:00:17.475444 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.475378 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:00:17.475444 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475387 2572 server.go:87] "Starting device plugin registration server" Nov 03 20:00:17.475610 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475598 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:00:17.475664 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475611 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:00:17.475711 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475702 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:00:17.475783 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475771 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:00:17.475845 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.475783 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:00:17.476299 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.476272 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:00:17.476364 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.476340 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:17.576302 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.576259 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.578085 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.577094 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.578085 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.577122 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.578085 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.577132 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.578085 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.577162 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.579800 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.579781 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:00:17.581113 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.581094 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:00:17.581192 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.581115 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:00:17.581192 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.581139 2572 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:00:17.581192 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.581168 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:00:17.589445 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.589219 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.589445 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.589270 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-81.us-east-2.compute.internal\": node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:17.589445 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.589236 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:17.631246 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.631225 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:17.681951 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.681909 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal"] Nov 03 20:00:17.682014 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.681988 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.682777 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.682761 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.682839 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.682789 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.682839 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.682799 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.684859 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.684846 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.685016 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685001 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.685054 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685042 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.685535 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685500 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.685535 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685527 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.685646 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685546 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.685646 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685550 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.685646 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685556 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.685646 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.685562 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.687493 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.687478 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.687587 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.687518 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:17.688248 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.688225 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:17.688345 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.688257 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:17.688345 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.688271 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:17.713840 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.713823 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.716645 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.716626 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.717913 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.717894 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.717975 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.717922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.717975 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.717938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7974090a1f6621f0627618a748dcd304-config\") pod \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"7974090a1f6621f0627618a748dcd304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.731938 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.731921 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:17.818244 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.818220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7974090a1f6621f0627618a748dcd304-config\") pod \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"7974090a1f6621f0627618a748dcd304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.818334 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.818245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.818334 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.818263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.818334 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.818313 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.818334 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.818315 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7974090a1f6621f0627618a748dcd304-config\") pod \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"7974090a1f6621f0627618a748dcd304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.818468 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:17.818333 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:17.832371 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.832327 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:17.933053 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:17.933031 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.017307 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.017289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:18.019480 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.019463 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:18.034057 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.033919 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.134478 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.134427 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.234892 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.234870 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.335394 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.335372 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.353560 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.353536 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:00:18.353689 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.353671 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:00:18.424374 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.424324 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:18.436127 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.436104 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.465306 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.465247 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:55:17 +0000 UTC" deadline="2028-06-06 13:24:05.473199442 +0000 UTC" Nov 03 20:00:18.465306 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.465304 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13937h23m47.007897809s" Nov 03 20:00:18.536900 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.536871 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.618903 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:18.618878 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7974090a1f6621f0627618a748dcd304.slice/crio-5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e WatchSource:0}: Error finding container 5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e: Status 404 returned error can't find the container with id 5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e Nov 03 20:00:18.622013 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.621999 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:18.622992 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.622974 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:18.623069 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.623014 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:18.623069 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.623030 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:18.624952 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.624940 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:00:18.637135 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.637119 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.737686 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:18.737667 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:00:18.788465 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.788450 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:18.791236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.791218 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:18.792720 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:18.792701 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70b0209b579d0e0dc902659c53a82cd.slice/crio-ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1 WatchSource:0}: Error finding container ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1: Status 404 returned error can't find the container with id ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1 Nov 03 20:00:18.816801 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.816784 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:18.831699 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.831683 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:00:18.832843 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.832832 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:00:18.840465 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:18.840452 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:00:19.168366 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.168344 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:19.395356 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.395287 2572 apiserver.go:52] "Watching apiserver" Nov 03 20:00:19.404935 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.404892 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:00:19.405317 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.405295 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bpqrl","kube-system/global-pull-secret-syncer-6k5bk","kube-system/konnectivity-agent-cg74r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-kvv5k","openshift-cluster-node-tuning-operator/tuned-htdgg","openshift-network-diagnostics/network-check-target-kglzd","openshift-network-operator/iptables-alerter-q4r6s","openshift-ovn-kubernetes/ovnkube-node-dt8zf","kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw","openshift-dns/node-resolver-j4p9c","openshift-image-registry/node-ca-n27x8","openshift-multus/multus-88mp6"] Nov 03 20:00:19.408820 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.408168 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.410702 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.410669 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.410804 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.410744 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:19.415112 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.413710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.415112 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.414752 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:00:19.415112 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.414829 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.415112 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.414869 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.415112 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.415084 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kmt87\"" Nov 03 20:00:19.415416 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.415336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:19.415416 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.415398 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:19.416530 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.415934 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:00:19.423953 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.422990 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.425449 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.425433 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.425938 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.425917 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:19.426093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bfhg\" (UniqueName: \"kubernetes.io/projected/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-kube-api-access-9bfhg\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.426220 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.426398 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.426398 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-host\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.426398 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426341 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-serviceca\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.426398 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426390 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-kubelet-config\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.426692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-dbus\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.426692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426461 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-agent-certs\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.426692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-cnibin\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.426692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.426692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.426692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qz2s\" (UniqueName: \"kubernetes.io/projected/a5c72c64-5332-4820-870d-8a38037b2604-kube-api-access-6qz2s\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.426944 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-konnectivity-ca\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.426944 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426737 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtlv\" (UniqueName: \"kubernetes.io/projected/757e39d9-7832-46de-8c68-930c0b80943e-kube-api-access-wqtlv\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:19.426944 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-system-cni-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.426944 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.426830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.428246 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.428224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:19.428347 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.428315 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:19.431373 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.431352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.433677 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.433656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.436215 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.436133 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.439345 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.439324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.439519 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.439368 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:00:19.442051 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.442028 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.461786 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.461758 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlwtb" Nov 03 20:00:19.475210 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.474870 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wlwtb" Nov 03 20:00:19.518455 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.518432 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:00:19.523086 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.522793 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.523086 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.522978 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.523086 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.523011 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.523086 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.523058 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:00:19.523364 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.523213 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.524206 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524112 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:00:19.524206 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524137 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524441 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524471 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524496 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524586 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524599 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524635 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lxhjg\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524667 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524718 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vlwgf\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524720 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:00:19.524769 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.524749 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:00:19.525320 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.525156 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:00:19.525320 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.525209 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:00:19.525844 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.525826 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.525929 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.525834 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.525929 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.525835 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfqpt\"" Nov 03 20:00:19.525929 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.525896 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8fc4f\"" Nov 03 20:00:19.526108 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526090 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:00:19.526165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526103 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbl6p\"" Nov 03 20:00:19.526165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526151 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:00:19.526165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526156 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:00:19.526311 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526175 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnml9\"" Nov 03 20:00:19.526311 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526202 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s55dl\"" Nov 03 20:00:19.526311 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526207 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:00:19.526529 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526509 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:00:19.526655 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.526585 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whl79\"" Nov 03 20:00:19.527741 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-cnibin\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.527825 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qz2s\" (UniqueName: \"kubernetes.io/projected/a5c72c64-5332-4820-870d-8a38037b2604-kube-api-access-6qz2s\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.527825 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-konnectivity-ca\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.527825 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527805 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-cnibin\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.527825 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527808 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-daemon-config\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.528045 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhsm\" (UniqueName: \"kubernetes.io/projected/82de9774-f285-46a2-a1d0-cf66b6f780b2-kube-api-access-blhsm\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.528045 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.528045 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527949 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-socket-dir-parent\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.528045 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-kubelet\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.528045 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.527999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-sys-fs\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.528045 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfhg\" (UniqueName: \"kubernetes.io/projected/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-kube-api-access-9bfhg\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-slash\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-ovn\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-script-lib\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-sys\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-host\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-host\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.528344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-serviceca\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mvpb\" (UniqueName: \"kubernetes.io/projected/05ab659c-a2dc-4764-a537-4b86fba538d2-kube-api-access-6mvpb\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-bin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-socket-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-kubelet\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528665 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysconfig\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.528733 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-kubernetes\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528747 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-systemd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-etc-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c8451-1aac-44aa-a685-28b03801682e-ovn-node-metrics-cert\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-modprobe-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528913 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-host\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.528991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-etc-selinux\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.529096 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529075 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-log-socket\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd55j\" (UniqueName: \"kubernetes.io/projected/849c8451-1aac-44aa-a685-28b03801682e-kube-api-access-hd55j\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529290 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-tmp\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529331 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-system-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-k8s-cni-cncf-io\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529384 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-konnectivity-ca\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-systemd-units\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529497 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-node-log\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.529563 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-conf\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529631 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-var-lib-kubelet\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529661 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da65b23-4da0-4d6e-8475-0a5bc3c19768-host-slash\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529676 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbwh\" (UniqueName: \"kubernetes.io/projected/1679fbb3-a322-4c15-9837-df1040cc3f01-kube-api-access-hgbwh\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529718 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtlv\" (UniqueName: \"kubernetes.io/projected/757e39d9-7832-46de-8c68-930c0b80943e-kube-api-access-wqtlv\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529788 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-etc-kubernetes\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-systemd\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4wn\" (UniqueName: \"kubernetes.io/projected/3da65b23-4da0-4d6e-8475-0a5bc3c19768-kube-api-access-2h4wn\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529977 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.529978 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.530033 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-registration-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.530125 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:00:20.030104356 +0000 UTC m=+3.026875555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:19.530274 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530173 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82de9774-f285-46a2-a1d0-cf66b6f780b2-tmp-dir\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-dbus\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-agent-certs\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-var-lib-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530347 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-serviceca\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-config\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530406 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-dbus\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-cnibin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.531032 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-hostroot\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.531528 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.531503 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:00:19.536897 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.530571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-bin\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.536897 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.536802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-run\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.536897 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.536856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-cni-binary-copy\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.537120 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.536909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-multus\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.537120 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.536960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-device-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.537120 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.536999 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-system-cni-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.537120 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537055 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-netns\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.537120 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-netd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.537120 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-env-overrides\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537143 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-agent-certs\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da65b23-4da0-4d6e-8475-0a5bc3c19768-iptables-alerter-script\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82de9774-f285-46a2-a1d0-cf66b6f780b2-hosts-file\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537243 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-system-cni-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537282 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537390 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-tuned\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.537448 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.537403 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-netns\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.537506 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:20.037485848 +0000 UTC m=+3.034257047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t4t9\" (UniqueName: \"kubernetes.io/projected/2df37093-9c54-4d41-ac8b-b41d8f83579d-kube-api-access-6t4t9\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-kubelet-config\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537672 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-lib-modules\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537702 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-kubelet-config\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-conf-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.538717 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.537787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-multus-certs\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.542200 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.542171 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qz2s\" (UniqueName: \"kubernetes.io/projected/a5c72c64-5332-4820-870d-8a38037b2604-kube-api-access-6qz2s\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.542706 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.542683 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfhg\" (UniqueName: \"kubernetes.io/projected/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-kube-api-access-9bfhg\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.545536 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.545510 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtlv\" (UniqueName: \"kubernetes.io/projected/757e39d9-7832-46de-8c68-930c0b80943e-kube-api-access-wqtlv\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:19.586838 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.586789 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"c70b0209b579d0e0dc902659c53a82cd","Type":"ContainerStarted","Data":"ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1"} Nov 03 20:00:19.588197 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.588169 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"7974090a1f6621f0627618a748dcd304","Type":"ContainerStarted","Data":"5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e"} Nov 03 20:00:19.638447 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-hostroot\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.638597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-bin\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.638597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-run\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.638597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638548 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-cni-binary-copy\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.638597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-multus\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638602 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-device-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-netns\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-netd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638675 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-env-overrides\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da65b23-4da0-4d6e-8475-0a5bc3c19768-iptables-alerter-script\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82de9774-f285-46a2-a1d0-cf66b6f780b2-hosts-file\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638790 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-tuned\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.638829 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-netns\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4t9\" (UniqueName: \"kubernetes.io/projected/2df37093-9c54-4d41-ac8b-b41d8f83579d-kube-api-access-6t4t9\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-lib-modules\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-conf-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-multus-certs\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638931 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-netd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-daemon-config\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638970 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-hostroot\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhsm\" (UniqueName: \"kubernetes.io/projected/82de9774-f285-46a2-a1d0-cf66b6f780b2-kube-api-access-blhsm\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.638996 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-bin\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-socket-dir-parent\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-kubelet\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-sys-fs\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639083 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-slash\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-ovn\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-script-lib\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-sys\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-host\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.639278 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvpb\" (UniqueName: \"kubernetes.io/projected/05ab659c-a2dc-4764-a537-4b86fba538d2-kube-api-access-6mvpb\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639212 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-bin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-socket-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639264 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-kubelet\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639312 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysconfig\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-kubernetes\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-systemd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639449 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-cni-binary-copy\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-etc-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639505 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-multus\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-etc-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639541 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-device-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c8451-1aac-44aa-a685-28b03801682e-ovn-node-metrics-cert\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639571 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-netns\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-modprobe-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.640180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639622 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-sys-fs\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-etc-selinux\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639683 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-log-socket\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd55j\" (UniqueName: \"kubernetes.io/projected/849c8451-1aac-44aa-a685-28b03801682e-kube-api-access-hd55j\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-tmp\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-system-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-k8s-cni-cncf-io\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-systemd-units\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-node-log\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-conf\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639950 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-var-lib-kubelet\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639963 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-env-overrides\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da65b23-4da0-4d6e-8475-0a5bc3c19768-host-slash\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640011 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbwh\" (UniqueName: \"kubernetes.io/projected/1679fbb3-a322-4c15-9837-df1040cc3f01-kube-api-access-hgbwh\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641046 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640039 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-etc-kubernetes\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-systemd\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4wn\" (UniqueName: \"kubernetes.io/projected/3da65b23-4da0-4d6e-8475-0a5bc3c19768-kube-api-access-2h4wn\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-registration-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640213 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82de9774-f285-46a2-a1d0-cf66b6f780b2-tmp-dir\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-var-lib-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-config\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640325 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da65b23-4da0-4d6e-8475-0a5bc3c19768-iptables-alerter-script\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640383 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-cnibin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640401 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-ovn\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640469 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82de9774-f285-46a2-a1d0-cf66b6f780b2-hosts-file\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640507 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.639030 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-run\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641043 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-lib-modules\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641096 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-netns\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-socket-dir-parent\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641354 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-kubelet\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.641874 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641381 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-conf-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641396 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-cnibin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-sys\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641492 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-systemd-units\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641514 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-host\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641671 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-script-lib\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641707 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-node-log\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da65b23-4da0-4d6e-8475-0a5bc3c19768-host-slash\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-var-lib-kubelet\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641783 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-bin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641822 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-multus-certs\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641848 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-registration-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641882 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-conf\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641901 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-socket-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641925 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-systemd\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.641983 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-etc-kubernetes\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642001 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-modprobe-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.642402 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642018 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-system-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642060 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642110 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-kubelet\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642152 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-etc-selinux\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysconfig\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642200 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-k8s-cni-cncf-io\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642234 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-systemd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642248 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-log-socket\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642269 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642294 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-kubernetes\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642297 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-var-lib-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.640413 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-slash\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642352 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.642636 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82de9774-f285-46a2-a1d0-cf66b6f780b2-tmp-dir\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.643055 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-config\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.643260 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.643095 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-tuned\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.644123 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.643570 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-daemon-config\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.644337 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.644313 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-tmp\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.644697 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.644672 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c8451-1aac-44aa-a685-28b03801682e-ovn-node-metrics-cert\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.653153 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.653085 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:19.653153 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.653116 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:19.653153 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.653131 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:19.653153 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:19.653198 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:20.153180451 +0000 UTC m=+3.149951645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:19.655411 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.655388 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbwh\" (UniqueName: \"kubernetes.io/projected/1679fbb3-a322-4c15-9837-df1040cc3f01-kube-api-access-hgbwh\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.655503 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.655405 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvpb\" (UniqueName: \"kubernetes.io/projected/05ab659c-a2dc-4764-a537-4b86fba538d2-kube-api-access-6mvpb\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.656345 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.656319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4t9\" (UniqueName: \"kubernetes.io/projected/2df37093-9c54-4d41-ac8b-b41d8f83579d-kube-api-access-6t4t9\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.656996 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.656977 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4wn\" (UniqueName: \"kubernetes.io/projected/3da65b23-4da0-4d6e-8475-0a5bc3c19768-kube-api-access-2h4wn\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.658164 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.658142 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhsm\" (UniqueName: \"kubernetes.io/projected/82de9774-f285-46a2-a1d0-cf66b6f780b2-kube-api-access-blhsm\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.659253 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.659226 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd55j\" (UniqueName: \"kubernetes.io/projected/849c8451-1aac-44aa-a685-28b03801682e-kube-api-access-hd55j\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.730455 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.730433 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:00:19.741491 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.741466 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:19.741996 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.741963 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f38eb4_b4e8_4673_a540_24c8ebe84ee8.slice/crio-808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4 WatchSource:0}: Error finding container 808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4: Status 404 returned error can't find the container with id 808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4 Nov 03 20:00:19.750942 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.750918 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b75f9f_c202_4a6c_b370_c5ce3d1de74c.slice/crio-1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a WatchSource:0}: Error finding container 1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a: Status 404 returned error can't find the container with id 1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a Nov 03 20:00:19.764501 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.764478 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:00:19.765647 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.765631 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:00:19.767807 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.767765 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:00:19.778675 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.778647 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ab659c_a2dc_4764_a537_4b86fba538d2.slice/crio-04a77a2f587972c793dd417baae473c9fce345562315db73746f00bbdabbd169 WatchSource:0}: Error finding container 04a77a2f587972c793dd417baae473c9fce345562315db73746f00bbdabbd169: Status 404 returned error can't find the container with id 04a77a2f587972c793dd417baae473c9fce345562315db73746f00bbdabbd169 Nov 03 20:00:19.800883 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.800857 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-88mp6" Nov 03 20:00:19.804979 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.804960 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:19.811892 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.811871 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:00:19.812380 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.812359 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1679fbb3_a322_4c15_9837_df1040cc3f01.slice/crio-94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07 WatchSource:0}: Error finding container 94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07: Status 404 returned error can't find the container with id 94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07 Nov 03 20:00:19.814186 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:19.814125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:00:19.816193 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.816154 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849c8451_1aac_44aa_a685_28b03801682e.slice/crio-05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22 WatchSource:0}: Error finding container 05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22: Status 404 returned error can't find the container with id 05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22 Nov 03 20:00:19.826026 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.825987 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df37093_9c54_4d41_ac8b_b41d8f83579d.slice/crio-6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793 WatchSource:0}: Error finding container 6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793: Status 404 returned error can't find the container with id 6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793 Nov 03 20:00:19.830195 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:19.830172 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82de9774_f285_46a2_a1d0_cf66b6f780b2.slice/crio-ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c WatchSource:0}: Error finding container ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c: Status 404 returned error can't find the container with id ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c Nov 03 20:00:20.043371 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.043329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:20.043564 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.043427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:20.043564 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.043507 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:20.043564 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.043548 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:20.043716 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.043594 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:00:21.043574047 +0000 UTC m=+4.040345244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:20.043716 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.043614 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:21.04360426 +0000 UTC m=+4.040375437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:20.244977 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.244936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:20.245160 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.245147 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:20.245223 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.245172 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:20.245223 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.245186 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:20.245324 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.245249 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:21.24522988 +0000 UTC m=+4.242001082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:20.485289 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.485199 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:19 +0000 UTC" deadline="2028-06-11 20:36:39.011149039 +0000 UTC" Nov 03 20:00:20.485289 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.485244 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14064h36m18.52590838s" Nov 03 20:00:20.582228 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.582149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:20.582400 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:20.582292 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:20.594738 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.594698 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j4p9c" event={"ID":"82de9774-f285-46a2-a1d0-cf66b6f780b2","Type":"ContainerStarted","Data":"ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c"} Nov 03 20:00:20.599111 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.599065 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" event={"ID":"2df37093-9c54-4d41-ac8b-b41d8f83579d","Type":"ContainerStarted","Data":"6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793"} Nov 03 20:00:20.601830 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.601803 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22"} Nov 03 20:00:20.605763 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.605736 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-88mp6" event={"ID":"1679fbb3-a322-4c15-9837-df1040cc3f01","Type":"ContainerStarted","Data":"94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07"} Nov 03 20:00:20.607909 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.607876 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q4r6s" event={"ID":"3da65b23-4da0-4d6e-8475-0a5bc3c19768","Type":"ContainerStarted","Data":"d3edb09c604bd7b56aa109dde93c7b54eaa04361d45ac9a599918194cc92b03f"} Nov 03 20:00:20.614361 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.613667 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-htdgg" event={"ID":"05ab659c-a2dc-4764-a537-4b86fba538d2","Type":"ContainerStarted","Data":"04a77a2f587972c793dd417baae473c9fce345562315db73746f00bbdabbd169"} Nov 03 20:00:20.616719 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.616446 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cg74r" event={"ID":"66b75f9f-c202-4a6c-b370-c5ce3d1de74c","Type":"ContainerStarted","Data":"1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a"} Nov 03 20:00:20.619382 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.619353 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerStarted","Data":"ed35a41a4635db4b29bb696539cc66db4f5aa2947d51d44ba736c2dc44971e0d"} Nov 03 20:00:20.621330 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.621222 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n27x8" event={"ID":"75f38eb4-b4e8-4673-a540-24c8ebe84ee8","Type":"ContainerStarted","Data":"808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4"} Nov 03 20:00:20.626275 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:20.625420 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"7974090a1f6621f0627618a748dcd304","Type":"ContainerStarted","Data":"1e9041bcf95320e2bfb06c672584685a9fff8ed4cad7e86ab2cf0a69ab7e8bda"} Nov 03 20:00:21.053759 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.052058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:21.053759 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.052114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:21.053759 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.052251 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:21.053759 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.052316 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:23.052298081 +0000 UTC m=+6.049069282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:21.053759 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.052798 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:21.053759 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.052855 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:00:23.052837419 +0000 UTC m=+6.049608602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:21.254397 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.254075 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:21.254566 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.254295 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:21.254566 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.254443 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:21.254566 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.254460 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:21.254566 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.254553 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:23.254532781 +0000 UTC m=+6.251303975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:21.485712 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.485669 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:19 +0000 UTC" deadline="2028-06-28 05:06:21.566315453 +0000 UTC" Nov 03 20:00:21.485712 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.485708 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14457h6m0.080610749s" Nov 03 20:00:21.585528 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.584768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:21.585528 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.584905 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:21.585528 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:21.585311 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:21.585528 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:21.585424 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:22.582340 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:22.581822 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:22.582340 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:22.581958 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:22.641630 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:22.641311 2572 generic.go:356] "Generic (PLEG): container finished" podID="c70b0209b579d0e0dc902659c53a82cd" containerID="af4ca698098c9dc4eee9927264849323d3c4b33ff0a03a5f900ffed288cf74bf" exitCode=0 Nov 03 20:00:22.641630 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:22.641361 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"c70b0209b579d0e0dc902659c53a82cd","Type":"ContainerDied","Data":"af4ca698098c9dc4eee9927264849323d3c4b33ff0a03a5f900ffed288cf74bf"} Nov 03 20:00:22.662805 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:22.662248 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" podStartSLOduration=3.6622281230000002 podStartE2EDuration="3.662228123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:20.644764825 +0000 UTC m=+3.641536027" watchObservedRunningTime="2026-11-03 20:00:22.662228123 +0000 UTC m=+5.658999327" Nov 03 20:00:23.068401 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.068369 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:23.068569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.068421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:23.068569 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.068561 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:23.068693 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.068623 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:27.068605384 +0000 UTC m=+10.065376567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:23.069044 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.069017 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:23.069137 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.069074 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:00:27.0690592 +0000 UTC m=+10.065830385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:23.269890 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.269825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:23.270059 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.270039 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:23.270143 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.270062 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:23.270143 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.270079 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:23.270249 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.270141 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:27.270122417 +0000 UTC m=+10.266893610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:23.586268 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.586203 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:23.586794 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.586571 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:23.586794 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.586665 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:23.586975 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:23.586890 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:23.650044 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.650005 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"c70b0209b579d0e0dc902659c53a82cd","Type":"ContainerStarted","Data":"93b6694059b8570f5ad5d114da9302e98725d172cfed095fe813024617badd99"} Nov 03 20:00:23.658207 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.658176 2572 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="a1536072e67912747fb2dcdb651087c06da274da344d14e7bfa8a9101a855a1f" exitCode=0 Nov 03 20:00:23.658383 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.658237 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerDied","Data":"a1536072e67912747fb2dcdb651087c06da274da344d14e7bfa8a9101a855a1f"} Nov 03 20:00:23.672901 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:23.672847 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" podStartSLOduration=4.672830029 podStartE2EDuration="4.672830029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:23.672014642 +0000 UTC m=+6.668785845" watchObservedRunningTime="2026-11-03 20:00:23.672830029 +0000 UTC m=+6.669601232" Nov 03 20:00:24.582607 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:24.582574 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:24.582773 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:24.582710 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:24.663513 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:24.663460 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cg74r" event={"ID":"66b75f9f-c202-4a6c-b370-c5ce3d1de74c","Type":"ContainerStarted","Data":"9fbcb63d5968a99307061386b79434c71af95d6521b73a867fbc527caca15afe"} Nov 03 20:00:24.760589 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:24.760556 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:24.761156 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:24.761133 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:24.779221 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:24.779143 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cg74r" podStartSLOduration=2.911158312 podStartE2EDuration="6.779125629s" totalImagesPullingTime="3.867967317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:24.682443041 +0000 UTC m=+7.679214244" watchObservedRunningTime="2026-11-03 20:00:24.779125629 +0000 UTC m=+7.775896831" Nov 03 20:00:25.585205 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:25.584521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:25.585205 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:25.584656 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:25.585205 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:25.585071 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:25.585205 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:25.585174 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:25.667821 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:25.667790 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" event={"ID":"2df37093-9c54-4d41-ac8b-b41d8f83579d","Type":"ContainerStarted","Data":"ee8c7c1a90adc4b8fc071a55907c696495fb0dc45ea7db3654df75e14dc4573b"} Nov 03 20:00:26.581879 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:26.581847 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:26.582045 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:26.581987 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:26.671469 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:26.671438 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:00:27.105888 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.105062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:27.105888 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.105127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:27.105888 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.105290 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:27.105888 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.105355 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:35.105336732 +0000 UTC m=+18.102107924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:27.105888 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.105794 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:27.105888 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.105851 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:00:35.105834003 +0000 UTC m=+18.102605202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:27.308321 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.307720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:27.308321 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.307930 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:27.308321 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.307951 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:27.308321 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.307965 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:27.308321 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.308016 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:35.307998637 +0000 UTC m=+18.304769830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:27.475271 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.475240 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:00:27.481613 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.480564 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:00:27.475264227Z","UUID":"6d768cb0-6c98-4dd8-9788-baa56a98aaa4","Handler":null,"Name":"","Endpoint":""} Nov 03 20:00:27.482690 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.482435 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:00:27.482690 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.482459 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:00:27.584930 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.584885 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:27.585113 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.585010 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:27.585342 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.585323 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:27.585464 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:27.585433 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:27.676767 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:27.676655 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" event={"ID":"2df37093-9c54-4d41-ac8b-b41d8f83579d","Type":"ContainerStarted","Data":"8de2b504f23f636e084f8057502a4495a47ba78daa0d2f66c5066e1bd5cc707d"} Nov 03 20:00:28.582137 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.582054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:28.582275 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:28.582192 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:28.683098 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.683049 2572 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="a121d8dce8715d13dfc017e2177b56e528163cd3aab6b84d026c4ab87eab5fc8" exitCode=0 Nov 03 20:00:28.683743 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.683719 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerDied","Data":"a121d8dce8715d13dfc017e2177b56e528163cd3aab6b84d026c4ab87eab5fc8"} Nov 03 20:00:28.686109 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.685782 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n27x8" event={"ID":"75f38eb4-b4e8-4673-a540-24c8ebe84ee8","Type":"ContainerStarted","Data":"0defad71e0ab76403921949befcddd466276594962214ad21b230c178ef63dbb"} Nov 03 20:00:28.687567 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.687538 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j4p9c" event={"ID":"82de9774-f285-46a2-a1d0-cf66b6f780b2","Type":"ContainerStarted","Data":"631e6266ab5aa696ab2a6d00d3d66957cd217e84f894faadba3d1033323f9522"} Nov 03 20:00:28.689468 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.689132 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-88mp6" event={"ID":"1679fbb3-a322-4c15-9837-df1040cc3f01","Type":"ContainerStarted","Data":"439f6cc1d369d7ebedcc2b7c6e39685a63a16dba594be795dd42837fe5951e9c"} Nov 03 20:00:28.731011 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.730943 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-88mp6" podStartSLOduration=1.947328201 podStartE2EDuration="10.730922078s" totalImagesPullingTime="8.783593877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:28.72961952 +0000 UTC m=+11.726390720" watchObservedRunningTime="2026-11-03 20:00:28.730922078 +0000 UTC m=+11.727693277" Nov 03 20:00:28.761455 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.761396 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j4p9c" podStartSLOduration=2.751937608 podStartE2EDuration="10.761378751s" totalImagesPullingTime="8.009441143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:28.760781125 +0000 UTC m=+11.757552329" watchObservedRunningTime="2026-11-03 20:00:28.761378751 +0000 UTC m=+11.758149953" Nov 03 20:00:28.797044 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:28.796992 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n27x8" podStartSLOduration=2.752725549 podStartE2EDuration="10.796975255s" totalImagesPullingTime="8.044249706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:28.796846622 +0000 UTC m=+11.793617820" watchObservedRunningTime="2026-11-03 20:00:28.796975255 +0000 UTC m=+11.793746439" Nov 03 20:00:29.582182 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:29.582149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:29.582423 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:29.582196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:29.582423 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:29.582304 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:29.582564 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:29.582418 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:30.581863 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.581671 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:30.582314 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:30.581951 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:30.693982 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.693860 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q4r6s" event={"ID":"3da65b23-4da0-4d6e-8475-0a5bc3c19768","Type":"ContainerStarted","Data":"084f707c9465d29a53ee76c9aad1bed1d238165327c4bfe87af4347d93c09b87"} Nov 03 20:00:30.696160 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.696132 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-htdgg" event={"ID":"05ab659c-a2dc-4764-a537-4b86fba538d2","Type":"ContainerStarted","Data":"2fe6ead9dab0bdd7853bca21b6728394e7281a197de6cd0078c2d2f205dc7e96"} Nov 03 20:00:30.698550 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.698524 2572 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="d628f356bcb9a14b06df6b67cee2c88bff87b41a66b340846be40ca40ff4928d" exitCode=0 Nov 03 20:00:30.698651 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.698613 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerDied","Data":"d628f356bcb9a14b06df6b67cee2c88bff87b41a66b340846be40ca40ff4928d"} Nov 03 20:00:30.711532 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.711343 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q4r6s" podStartSLOduration=4.243331089 podStartE2EDuration="12.711332488s" totalImagesPullingTime="8.468001399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:30.710604042 +0000 UTC m=+13.707375239" watchObservedRunningTime="2026-11-03 20:00:30.711332488 +0000 UTC m=+13.708103686" Nov 03 20:00:30.756954 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:30.756907 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-htdgg" podStartSLOduration=2.356930497 podStartE2EDuration="12.756891358s" totalImagesPullingTime="10.399960861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:30.75634169 +0000 UTC m=+13.753112898" watchObservedRunningTime="2026-11-03 20:00:30.756891358 +0000 UTC m=+13.753662555" Nov 03 20:00:31.584458 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:31.584425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:31.585109 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:31.584544 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:31.585109 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:31.584585 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:31.585109 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:31.584677 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:31.705842 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:31.705818 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" event={"ID":"2df37093-9c54-4d41-ac8b-b41d8f83579d","Type":"ContainerStarted","Data":"d34df26e731159aaee3bca1e1c04c2ce8edde2e24f063de0bf7be508d6cd322d"} Nov 03 20:00:31.726908 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:31.726866 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" podStartSLOduration=3.404320023 podStartE2EDuration="13.726854049s" totalImagesPullingTime="10.322534026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:31.726805075 +0000 UTC m=+14.723576274" watchObservedRunningTime="2026-11-03 20:00:31.726854049 +0000 UTC m=+14.723625247" Nov 03 20:00:32.581320 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:32.581289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:32.581460 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:32.581393 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:32.709370 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:32.709332 2572 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="9818d7b0901142a305d2100818fdbc2c7f2490c90ebe2132e9d5bae6a06209a1" exitCode=0 Nov 03 20:00:32.709962 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:32.709434 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerDied","Data":"9818d7b0901142a305d2100818fdbc2c7f2490c90ebe2132e9d5bae6a06209a1"} Nov 03 20:00:33.582131 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:33.581921 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:33.582297 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:33.581969 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:33.582297 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:33.582213 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:33.582391 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:33.582292 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:34.582240 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:34.582208 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:34.582648 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:34.582323 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:35.165425 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.165354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:35.165425 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.165393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:35.165603 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.165479 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:35.165603 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.165484 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:35.165603 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.165546 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:51.165534334 +0000 UTC m=+34.162305511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:35.165603 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.165560 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:00:51.165554597 +0000 UTC m=+34.162325773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:35.366708 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.366680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:35.366859 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.366842 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:35.366895 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.366865 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:35.366895 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.366875 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:35.366962 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.366924 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:51.366909175 +0000 UTC m=+34.363680366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:35.581999 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.581972 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:35.582133 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.582064 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:35.582188 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.582140 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:35.582246 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:35.582229 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:35.717754 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.717724 2572 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="07b2202d1158d8e7ed0c74cd50129f0634c53d48dddb806c62daa355d80f546c" exitCode=0 Nov 03 20:00:35.718099 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:35.717772 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerDied","Data":"07b2202d1158d8e7ed0c74cd50129f0634c53d48dddb806c62daa355d80f546c"} Nov 03 20:00:36.582297 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:36.582267 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:36.582455 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:36.582390 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:36.722288 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:36.722256 2572 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="2ca7cc5f999570d4ec7eb09ac690deda64fac08e065706ca16ab6124566dc13d" exitCode=0 Nov 03 20:00:36.722667 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:36.722300 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerDied","Data":"2ca7cc5f999570d4ec7eb09ac690deda64fac08e065706ca16ab6124566dc13d"} Nov 03 20:00:37.583145 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:37.582652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:37.583145 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:37.582783 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:37.583145 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:37.582838 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:37.583145 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:37.582972 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:37.728971 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:37.728936 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" event={"ID":"a5c72c64-5332-4820-870d-8a38037b2604","Type":"ContainerStarted","Data":"5265dc9d4ac04be6e9ebb769c44d4a6e8f862821c90f5f80c8306622ded6bff9"} Nov 03 20:00:37.756255 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:37.756208 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kvv5k" podStartSLOduration=8.179396278 podStartE2EDuration="19.756192352s" totalImagesPullingTime="11.576796074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:37.755860256 +0000 UTC m=+20.752631453" watchObservedRunningTime="2026-11-03 20:00:37.756192352 +0000 UTC m=+20.752963550" Nov 03 20:00:38.581909 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:38.581875 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:38.582082 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:38.581984 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:39.581831 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:39.581796 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:39.581831 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:39.581836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:39.582380 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:39.581926 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:39.582380 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:39.582095 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:40.581422 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:40.581393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:40.581689 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:40.581491 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:41.582447 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.582272 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:41.582936 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.582336 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:41.582936 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:41.582557 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:41.582936 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:41.582603 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:41.740688 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.740667 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:00:41.740994 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.740971 2572 generic.go:356] "Generic (PLEG): container finished" podID="849c8451-1aac-44aa-a685-28b03801682e" containerID="5ff47328f0b3a6fee0960dedc7cd8fb4a3411888c8860cc70114e6b41eaf7045" exitCode=1 Nov 03 20:00:41.741048 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.741021 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"e46ae48288c304b25277792c7b7f75cd4dfe32fa24e04e41f6074a1d2a2f8f67"} Nov 03 20:00:41.741081 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.741073 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"c2d7fcfda92a7c1639843e85ef093cf956e6bda97fbef59756809abebe3b4264"} Nov 03 20:00:41.741113 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.741102 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"1272c8d0d21e1f6a69745ea732b32642260a1edf971fdb97bae10af15ec7470a"} Nov 03 20:00:41.741144 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.741129 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"97ee76587224dcd890905a5047d79a0a229ba9f072b5fa73048ed28f7a71fb80"} Nov 03 20:00:41.741175 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.741156 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerDied","Data":"5ff47328f0b3a6fee0960dedc7cd8fb4a3411888c8860cc70114e6b41eaf7045"} Nov 03 20:00:41.741205 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:41.741183 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"10a7cdcc8f1365baaca13cebb1cf59947d86e48cf54227a144aa4abe52d73116"} Nov 03 20:00:42.315039 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:42.315009 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:42.315221 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:42.315126 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:00:42.315551 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:42.315531 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:00:42.582113 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:42.582060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:42.582212 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:42.582140 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:43.581996 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:43.581918 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:43.581996 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:43.581938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:43.582543 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:43.582021 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:43.582543 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:43.582086 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:43.748592 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:43.748568 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:00:43.748913 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:43.748890 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"8cc2339e71e8169776632821326c9ba551dac15404f8f1769c15f6b2ae39f62d"} Nov 03 20:00:44.581800 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:44.581765 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:44.581971 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:44.581877 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:45.582265 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.582188 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:45.583063 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:45.582314 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:45.583063 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.582359 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:45.583063 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:45.582437 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:45.756373 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.756347 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:00:45.756744 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.756724 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"e9c0255d5a3145b367043c41fdb60a9778745f5a0ee1a82c1e7fb0b5c172ced5"} Nov 03 20:00:45.757028 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.757006 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:45.757102 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.757089 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:45.757160 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.757144 2572 scope.go:122] "RemoveContainer" containerID="5ff47328f0b3a6fee0960dedc7cd8fb4a3411888c8860cc70114e6b41eaf7045" Nov 03 20:00:45.773131 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:45.773110 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:46.581849 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.581818 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:46.582032 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:46.581940 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:46.763101 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.763074 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:00:46.763557 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.763531 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" event={"ID":"849c8451-1aac-44aa-a685-28b03801682e","Type":"ContainerStarted","Data":"66812bcd3be0234f9e8885487c3f70d19456742f4ac7ce4efdffca9ea77b2256"} Nov 03 20:00:46.763712 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.763699 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:46.778217 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.778196 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:00:46.800435 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.800397 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" podStartSLOduration=7.878319081 podStartE2EDuration="28.800386041s" totalImagesPullingTime="20.92206696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:46.798754079 +0000 UTC m=+29.795525291" watchObservedRunningTime="2026-11-03 20:00:46.800386041 +0000 UTC m=+29.797157254" Nov 03 20:00:46.976682 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.976654 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6k5bk"] Nov 03 20:00:46.976848 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.976802 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:46.976936 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:46.976890 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:46.979743 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.979696 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpqrl"] Nov 03 20:00:46.979859 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.979790 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:46.979915 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:46.979867 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:46.982299 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.982279 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kglzd"] Nov 03 20:00:46.982430 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:46.982356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:46.982430 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:46.982414 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:48.582464 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:48.582265 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:48.582861 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:48.582314 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:48.582861 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:48.582559 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:48.582861 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:48.582602 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:48.582861 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:48.582316 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:48.582861 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:48.582694 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:50.581446 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:50.581413 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:50.581931 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:50.581528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:50.581931 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:50.581548 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:50.581931 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:50.581561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:50.581931 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:50.581651 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:50.581931 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:50.581751 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:51.178186 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:51.178154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:51.178342 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:51.178190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:51.178342 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.178294 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:51.178342 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.178308 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:51.178471 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.178346 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret podName:3b7ef6fe-524b-4542-b281-27f7fce2deb8 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:23.178330218 +0000 UTC m=+66.175101397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret") pod "global-pull-secret-syncer-6k5bk" (UID: "3b7ef6fe-524b-4542-b281-27f7fce2deb8") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:51.178471 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.178370 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs podName:757e39d9-7832-46de-8c68-930c0b80943e nodeName:}" failed. No retries permitted until 2026-11-03 20:01:23.178353142 +0000 UTC m=+66.175124335 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs") pod "network-metrics-daemon-bpqrl" (UID: "757e39d9-7832-46de-8c68-930c0b80943e") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:51.380400 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:51.380379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:51.380527 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.380499 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:51.380568 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.380533 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:51.380568 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.380542 2572 projected.go:196] Error preparing data for projected volume kube-api-access-nnl8s for pod openshift-network-diagnostics/network-check-target-kglzd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:51.380627 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:51.380587 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s podName:7978e0e4-9979-42cf-a15b-767a03352919 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:23.380575094 +0000 UTC m=+66.377346283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nnl8s" (UniqueName: "kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s") pod "network-check-target-kglzd" (UID: "7978e0e4-9979-42cf-a15b-767a03352919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:52.581836 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:52.581810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:52.581836 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:52.581838 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:52.582298 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:52.581898 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bpqrl" podUID="757e39d9-7832-46de-8c68-930c0b80943e" Nov 03 20:00:52.582298 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:52.581947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:52.582298 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:52.582052 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6k5bk" podUID="3b7ef6fe-524b-4542-b281-27f7fce2deb8" Nov 03 20:00:52.582298 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:52.582142 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kglzd" podUID="7978e0e4-9979-42cf-a15b-767a03352919" Nov 03 20:00:53.310443 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.310386 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeReady" Nov 03 20:00:53.310603 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.310489 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:00:53.348669 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.348638 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7qnz8"] Nov 03 20:00:53.396059 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.396034 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6kg75"] Nov 03 20:00:53.396333 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.396315 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:00:53.401759 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.401736 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.402228 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.402203 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.412219 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.412199 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fbn7r"] Nov 03 20:00:53.412319 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.412288 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:00:53.426334 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.426317 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-hft4t"] Nov 03 20:00:53.426446 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.426432 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:00:53.440567 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.440492 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx"] Nov 03 20:00:53.440567 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.440568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.444637 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.444617 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:00:53.444882 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.444869 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xfnff\"" Nov 03 20:00:53.454583 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.454568 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2"] Nov 03 20:00:53.454694 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.454678 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.470804 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.470787 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2"] Nov 03 20:00:53.470907 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.470895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.474905 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.474888 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph"] Nov 03 20:00:53.475068 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.475049 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.476708 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.476691 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mxp5k"] Nov 03 20:00:53.476804 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.476788 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.492574 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.492555 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-585db4bb4c-xs84v"] Nov 03 20:00:53.492574 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.492567 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.494617 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.494600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7rm\" (UniqueName: \"kubernetes.io/projected/148af517-1bbd-404e-97da-cadd491b1ded-kube-api-access-6m7rm\") pod \"kas-connection-checker-685895c7c9-7qnz8\" (UID: \"148af517-1bbd-404e-97da-cadd491b1ded\") " pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:00:53.494683 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.494671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6fx\" (UniqueName: \"kubernetes.io/projected/1551a4b4-7bd1-47ec-ae80-e79f4262761e-kube-api-access-9r6fx\") pod \"kas-connection-checker-685895c7c9-6kg75\" (UID: \"1551a4b4-7bd1-47ec-ae80-e79f4262761e\") " pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:00:53.502608 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502588 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.502823 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502804 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.502895 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502846 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.502895 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502869 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:00:53.502895 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502877 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.502895 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502895 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:00:53.503093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502910 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.503093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.502927 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.503093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503055 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.503093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503057 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.503093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503062 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.503093 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503075 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:00:53.503294 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503104 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:00:53.503294 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503159 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:00:53.503294 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.503193 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.504411 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.504399 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.504467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.504428 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.510545 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.510529 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:00:53.510634 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.510621 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh"] Nov 03 20:00:53.510678 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.510670 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.514686 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.514668 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:00:53.514821 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.514799 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:00:53.514980 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.514964 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.515059 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.514986 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.515095 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.515068 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:00:53.528756 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.528733 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68844ff664-xnjc5"] Nov 03 20:00:53.528842 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.528828 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:00:53.545317 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545290 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:00:53.545433 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545416 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:00:53.545479 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545432 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:00:53.545641 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545625 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rfsd7\"" Nov 03 20:00:53.545729 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545644 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:00:53.545729 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545691 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:00:53.545729 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545694 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:00:53.545729 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545693 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:00:53.545729 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545696 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:00:53.545729 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545727 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f6jr5\"" Nov 03 20:00:53.545967 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545741 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-458rx\"" Nov 03 20:00:53.545967 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545786 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zh9fq\"" Nov 03 20:00:53.545967 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.545839 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-grxf8\"" Nov 03 20:00:53.548956 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.546251 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vncq9\"" Nov 03 20:00:53.548956 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.546264 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ms2v5\"" Nov 03 20:00:53.549336 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.549318 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d87675b69-6m79x"] Nov 03 20:00:53.549471 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.549455 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.553276 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.553262 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:00:53.553535 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.553496 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:00:53.553622 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.553566 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pw8jf\"" Nov 03 20:00:53.553622 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.553596 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:00:53.563209 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.563168 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zwgdc"] Nov 03 20:00:53.563286 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.563274 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.575745 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.575730 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll"] Nov 03 20:00:53.575845 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.575833 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:53.595263 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-config\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.595263 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-tmp\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kckq\" (UniqueName: \"kubernetes.io/projected/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-kube-api-access-7kckq\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfsd\" (UniqueName: \"kubernetes.io/projected/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-kube-api-access-qqfsd\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595400 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9kl\" (UniqueName: \"kubernetes.io/projected/837adf32-c68a-46ef-bca6-5b5364f74576-kube-api-access-2z9kl\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595464 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-trusted-ca\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.595569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xvc\" (UniqueName: \"kubernetes.io/projected/844916e1-ce51-4692-907d-f30d1b6a9eb8-kube-api-access-w7xvc\") pod \"kas-connection-checker-685895c7c9-fbn7r\" (UID: \"844916e1-ce51-4692-907d-f30d1b6a9eb8\") " pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-snapshots\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-serving-cert\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595613 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-default-certificate\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-service-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.595773 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4fl\" (UniqueName: \"kubernetes.io/projected/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-kube-api-access-zp4fl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.595964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.595964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595824 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6fx\" (UniqueName: \"kubernetes.io/projected/1551a4b4-7bd1-47ec-ae80-e79f4262761e-kube-api-access-9r6fx\") pod \"kas-connection-checker-685895c7c9-6kg75\" (UID: \"1551a4b4-7bd1-47ec-ae80-e79f4262761e\") " pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:00:53.595964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-config\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.595964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7rm\" (UniqueName: \"kubernetes.io/projected/148af517-1bbd-404e-97da-cadd491b1ded-kube-api-access-6m7rm\") pod \"kas-connection-checker-685895c7c9-7qnz8\" (UID: \"148af517-1bbd-404e-97da-cadd491b1ded\") " pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:00:53.595964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-serving-cert\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.595964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjlq\" (UniqueName: \"kubernetes.io/projected/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-kube-api-access-vxjlq\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.596165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.595987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.596165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-stats-auth\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.596165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596052 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.596165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57sb\" (UniqueName: \"kubernetes.io/projected/333ef57f-4887-4c8f-81c8-962265ad3be0-kube-api-access-j57sb\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.596300 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86b25\" (UniqueName: \"kubernetes.io/projected/44590efe-c957-4995-8139-5b54567e51fd-kube-api-access-86b25\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.596300 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44590efe-c957-4995-8139-5b54567e51fd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.596300 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.596300 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.596240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/333ef57f-4887-4c8f-81c8-962265ad3be0-serving-cert\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.599877 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.599863 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7qnz8"] Nov 03 20:00:53.599928 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.599888 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fbn7r"] Nov 03 20:00:53.599974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.599961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:00:53.602536 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.602501 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.602536 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.602525 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.602681 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.602530 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:00:53.604106 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.604091 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-hft4t"] Nov 03 20:00:53.604177 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.604125 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2"] Nov 03 20:00:53.604177 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.604141 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx"] Nov 03 20:00:53.604177 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.604155 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll"] Nov 03 20:00:53.604177 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.604169 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph"] Nov 03 20:00:53.604318 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.604189 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mt748"] Nov 03 20:00:53.606314 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.606297 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.606399 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.606384 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.606458 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.606408 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:00:53.610059 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.610034 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7rm\" (UniqueName: \"kubernetes.io/projected/148af517-1bbd-404e-97da-cadd491b1ded-kube-api-access-6m7rm\") pod \"kas-connection-checker-685895c7c9-7qnz8\" (UID: \"148af517-1bbd-404e-97da-cadd491b1ded\") " pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:00:53.610140 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.610066 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6fx\" (UniqueName: \"kubernetes.io/projected/1551a4b4-7bd1-47ec-ae80-e79f4262761e-kube-api-access-9r6fx\") pod \"kas-connection-checker-685895c7c9-6kg75\" (UID: \"1551a4b4-7bd1-47ec-ae80-e79f4262761e\") " pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:00:53.621930 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.621914 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d6zdd"] Nov 03 20:00:53.622031 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.622017 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.645378 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.645360 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:00:53.645463 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.645411 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:00:53.645501 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.645486 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lrvk5\"" Nov 03 20:00:53.645588 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.645573 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-r5sx5\"" Nov 03 20:00:53.645621 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.645604 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5fnjm\"" Nov 03 20:00:53.646180 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646166 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6kg75"] Nov 03 20:00:53.646210 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646194 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mxp5k"] Nov 03 20:00:53.646210 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646209 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d87675b69-6m79x"] Nov 03 20:00:53.646264 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646228 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-585db4bb4c-xs84v"] Nov 03 20:00:53.646264 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646248 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh"] Nov 03 20:00:53.646316 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646269 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mt748"] Nov 03 20:00:53.646316 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646287 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d6zdd"] Nov 03 20:00:53.646316 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.646316 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646311 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2"] Nov 03 20:00:53.646420 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646332 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zwgdc"] Nov 03 20:00:53.646420 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.646349 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68844ff664-xnjc5"] Nov 03 20:00:53.650517 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.650492 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\"" Nov 03 20:00:53.696965 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.696948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-config\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.697042 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.696973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.697042 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.696994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-serving-cert\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.697042 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697009 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.697042 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjlq\" (UniqueName: \"kubernetes.io/projected/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-kube-api-access-vxjlq\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.697178 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.697178 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-stats-auth\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.697178 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697160 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmtp\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.697282 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697178 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.697282 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697224 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.697282 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697262 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.697429 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57sb\" (UniqueName: \"kubernetes.io/projected/333ef57f-4887-4c8f-81c8-962265ad3be0-kube-api-access-j57sb\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.697429 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.697344 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.19732669 +0000 UTC m=+37.194097905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : configmap references non-existent config key: service-ca.crt Nov 03 20:00:53.697429 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b25\" (UniqueName: \"kubernetes.io/projected/44590efe-c957-4995-8139-5b54567e51fd-kube-api-access-86b25\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.697429 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44590efe-c957-4995-8139-5b54567e51fd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.697429 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697450 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbz8z\" (UniqueName: \"kubernetes.io/projected/0ed58f76-099a-498d-844d-0069904b507b-kube-api-access-qbz8z\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/333ef57f-4887-4c8f-81c8-962265ad3be0-serving-cert\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697578 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697638 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-config\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.697701 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-tmp\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kckq\" (UniqueName: \"kubernetes.io/projected/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-kube-api-access-7kckq\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697759 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4faac008-58e3-41b1-9f56-8a8760071532-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfsd\" (UniqueName: \"kubernetes.io/projected/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-kube-api-access-qqfsd\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9kl\" (UniqueName: \"kubernetes.io/projected/837adf32-c68a-46ef-bca6-5b5364f74576-kube-api-access-2z9kl\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-trusted-ca\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.697966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.698006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.698050 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.698049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.698077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.698087 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-config\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.698104 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.698207 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.698256 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls podName:02c8db60-a588-40e9-a3b7-ba1ac37a5d47 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.198241966 +0000 UTC m=+37.195013143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mbgph" (UID: "02c8db60-a588-40e9-a3b7-ba1ac37a5d47") : secret "samples-operator-tls" not found Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.698398 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:53.698597 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.698461 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls podName:44590efe-c957-4995-8139-5b54567e51fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.198445919 +0000 UTC m=+37.195217118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-txxdx" (UID: "44590efe-c957-4995-8139-5b54567e51fd") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:53.699144 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.698546 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-tmp\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.699628 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.699608 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-trusted-ca\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.700705 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700036 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.700705 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xvc\" (UniqueName: \"kubernetes.io/projected/844916e1-ce51-4692-907d-f30d1b6a9eb8-kube-api-access-w7xvc\") pod \"kas-connection-checker-685895c7c9-fbn7r\" (UID: \"844916e1-ce51-4692-907d-f30d1b6a9eb8\") " pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:00:53.700705 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700452 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:53.700705 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700635 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44590efe-c957-4995-8139-5b54567e51fd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.700705 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700638 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-config\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-snapshots\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-serving-cert\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.700909 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-serving-cert\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701543 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcm6\" (UniqueName: \"kubernetes.io/projected/60d3aaa0-ccf0-42ae-92d5-44512e33aed4-kube-api-access-pjcm6\") pod \"volume-data-source-validator-58c9f5f46d-4gnrh\" (UID: \"60d3aaa0-ccf0-42ae-92d5-44512e33aed4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-default-certificate\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701660 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-service-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4fl\" (UniqueName: \"kubernetes.io/projected/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-kube-api-access-zp4fl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed58f76-099a-498d-844d-0069904b507b-cert\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-snapshots\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.702483 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.701835 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55szz\" (UniqueName: \"kubernetes.io/projected/9cb73706-3f89-4a36-83fa-8e5b42d55175-kube-api-access-55szz\") pod \"network-check-source-6f78d59f99-6tmll\" (UID: \"9cb73706-3f89-4a36-83fa-8e5b42d55175\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.701942 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-stats-auth\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.702419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.702579 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.702680 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.702753 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/333ef57f-4887-4c8f-81c8-962265ad3be0-serving-cert\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.702780 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:00:53.703236 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.702900 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.20288077 +0000 UTC m=+37.199651947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : secret "router-metrics-certs-default" not found Nov 03 20:00:53.704017 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.703991 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-service-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.705653 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.705624 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.706471 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.706352 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-default-certificate\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.707481 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.707457 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-serving-cert\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.709378 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.709357 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57sb\" (UniqueName: \"kubernetes.io/projected/333ef57f-4887-4c8f-81c8-962265ad3be0-kube-api-access-j57sb\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.709455 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.709439 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjlq\" (UniqueName: \"kubernetes.io/projected/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-kube-api-access-vxjlq\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.712290 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.712254 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b25\" (UniqueName: \"kubernetes.io/projected/44590efe-c957-4995-8139-5b54567e51fd-kube-api-access-86b25\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:53.712561 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.712539 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9kl\" (UniqueName: \"kubernetes.io/projected/837adf32-c68a-46ef-bca6-5b5364f74576-kube-api-access-2z9kl\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:53.714368 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.714349 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfsd\" (UniqueName: \"kubernetes.io/projected/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-kube-api-access-qqfsd\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.714458 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.714435 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xvc\" (UniqueName: \"kubernetes.io/projected/844916e1-ce51-4692-907d-f30d1b6a9eb8-kube-api-access-w7xvc\") pod \"kas-connection-checker-685895c7c9-fbn7r\" (UID: \"844916e1-ce51-4692-907d-f30d1b6a9eb8\") " pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:00:53.715467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.715447 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kckq\" (UniqueName: \"kubernetes.io/projected/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-kube-api-access-7kckq\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:53.715549 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.715486 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4fl\" (UniqueName: \"kubernetes.io/projected/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-kube-api-access-zp4fl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.721546 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.721526 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:00:53.732867 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.732851 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:00:53.745467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.745455 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:00:53.747566 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.747551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:00:53.761710 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.761693 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:00:53.793536 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.793004 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:00:53.802739 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802707 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcm6\" (UniqueName: \"kubernetes.io/projected/60d3aaa0-ccf0-42ae-92d5-44512e33aed4-kube-api-access-pjcm6\") pod \"volume-data-source-validator-58c9f5f46d-4gnrh\" (UID: \"60d3aaa0-ccf0-42ae-92d5-44512e33aed4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:00:53.802858 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed58f76-099a-498d-844d-0069904b507b-cert\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.802858 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802790 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-tmp-dir\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.802858 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55szz\" (UniqueName: \"kubernetes.io/projected/9cb73706-3f89-4a36-83fa-8e5b42d55175-kube-api-access-55szz\") pod \"network-check-source-6f78d59f99-6tmll\" (UID: \"9cb73706-3f89-4a36-83fa-8e5b42d55175\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmtp\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.802986 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803015 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803057 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-config-volume\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbz8z\" (UniqueName: \"kubernetes.io/projected/0ed58f76-099a-498d-844d-0069904b507b-kube-api-access-qbz8z\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.803115 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803112 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76vc\" (UniqueName: \"kubernetes.io/projected/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-kube-api-access-f76vc\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803196 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803227 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4faac008-58e3-41b1-9f56-8a8760071532-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803440 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:53.803538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803916 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.803916 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.803909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.804972 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.804951 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.805607 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.805535 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.806211 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.806184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.806643 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.806620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.807367 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.807343 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:53.807444 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.807370 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68844ff664-xnjc5: secret "image-registry-tls" not found Nov 03 20:00:53.807520 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.807475 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls podName:2f162f2a-0190-494e-9d6b-0ce6eefa90b5 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.307456382 +0000 UTC m=+37.304227746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls") pod "image-registry-68844ff664-xnjc5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5") : secret "image-registry-tls" not found Nov 03 20:00:53.807520 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.807495 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4faac008-58e3-41b1-9f56-8a8760071532-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:53.808063 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.808039 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.808374 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.808353 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:00:53.808445 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.808439 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert podName:4faac008-58e3-41b1-9f56-8a8760071532 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.308423762 +0000 UTC m=+37.305194955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zwgdc" (UID: "4faac008-58e3-41b1-9f56-8a8760071532") : secret "networking-console-plugin-cert" not found Nov 03 20:00:53.808545 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.808528 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:53.808614 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.808549 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d87675b69-6m79x: secret "image-registry-tls" not found Nov 03 20:00:53.808614 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.808584 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls podName:b843f245-431e-40af-a566-a4fa62de8c39 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.308572501 +0000 UTC m=+37.305343691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls") pod "image-registry-5d87675b69-6m79x" (UID: "b843f245-431e-40af-a566-a4fa62de8c39") : secret "image-registry-tls" not found Nov 03 20:00:53.808614 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.808585 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.808969 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.808851 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.809047 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.808951 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.809105 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.809091 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed58f76-099a-498d-844d-0069904b507b-cert\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.810986 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.810964 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.811854 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.811831 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.818135 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.817099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcm6\" (UniqueName: \"kubernetes.io/projected/60d3aaa0-ccf0-42ae-92d5-44512e33aed4-kube-api-access-pjcm6\") pod \"volume-data-source-validator-58c9f5f46d-4gnrh\" (UID: \"60d3aaa0-ccf0-42ae-92d5-44512e33aed4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:00:53.818135 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.817905 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.818135 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.817961 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbz8z\" (UniqueName: \"kubernetes.io/projected/0ed58f76-099a-498d-844d-0069904b507b-kube-api-access-qbz8z\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:53.818135 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.818020 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmtp\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.818466 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.818411 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55szz\" (UniqueName: \"kubernetes.io/projected/9cb73706-3f89-4a36-83fa-8e5b42d55175-kube-api-access-55szz\") pod \"network-check-source-6f78d59f99-6tmll\" (UID: \"9cb73706-3f89-4a36-83fa-8e5b42d55175\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:00:53.820762 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.820733 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:53.822642 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.822595 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:53.840859 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.840262 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:00:53.860866 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.860578 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:00:53.891075 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.890817 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:00:53.904744 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.904399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-tmp-dir\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.904744 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.904447 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.904744 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.904500 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-config-volume\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.904744 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.904573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76vc\" (UniqueName: \"kubernetes.io/projected/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-kube-api-access-f76vc\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.904919 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.904748 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:00:53.904919 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:53.904820 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls podName:99e197f7-de69-4fb1-b6d8-e1ffaa6d4903 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:54.404801571 +0000 UTC m=+37.401572765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls") pod "dns-default-d6zdd" (UID: "99e197f7-de69-4fb1-b6d8-e1ffaa6d4903") : secret "dns-default-metrics-tls" not found Nov 03 20:00:53.905290 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.905222 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-tmp-dir\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.905638 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.905617 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-config-volume\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.920246 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.920219 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76vc\" (UniqueName: \"kubernetes.io/projected/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-kube-api-access-f76vc\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:53.971308 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:53.971047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:00:54.003321 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.003300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:00:54.064464 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.064370 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7qnz8"] Nov 03 20:00:54.067440 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.067419 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6kg75"] Nov 03 20:00:54.067922 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.067898 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148af517_1bbd_404e_97da_cadd491b1ded.slice/crio-c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7 WatchSource:0}: Error finding container c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7: Status 404 returned error can't find the container with id c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7 Nov 03 20:00:54.075644 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.075597 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fbn7r"] Nov 03 20:00:54.092595 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.092067 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844916e1_ce51_4692_907d_f30d1b6a9eb8.slice/crio-c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca WatchSource:0}: Error finding container c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca: Status 404 returned error can't find the container with id c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca Nov 03 20:00:54.113218 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.113197 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-hft4t"] Nov 03 20:00:54.116082 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.116057 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07a2feb_3648_4f19_9385_baeb5eeeaeeb.slice/crio-e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3 WatchSource:0}: Error finding container e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3: Status 404 returned error can't find the container with id e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3 Nov 03 20:00:54.132210 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.132106 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2"] Nov 03 20:00:54.142876 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.142713 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714f1a1a_a854_4a9c_a2f6_d07990fe9f0e.slice/crio-deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb WatchSource:0}: Error finding container deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb: Status 404 returned error can't find the container with id deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb Nov 03 20:00:54.154633 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.154428 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2"] Nov 03 20:00:54.159206 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.159175 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8370c5cf_f442_4e7d_b8ab_ea6ede18552b.slice/crio-3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9 WatchSource:0}: Error finding container 3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9: Status 404 returned error can't find the container with id 3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9 Nov 03 20:00:54.177411 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.177394 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mxp5k"] Nov 03 20:00:54.180736 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.180710 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333ef57f_4887_4c8f_81c8_962265ad3be0.slice/crio-ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf WatchSource:0}: Error finding container ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf: Status 404 returned error can't find the container with id ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf Nov 03 20:00:54.193418 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.193392 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh"] Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.206490 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d3aaa0_ccf0_42ae_92d5_44512e33aed4.slice/crio-e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5 WatchSource:0}: Error finding container e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5: Status 404 returned error can't find the container with id e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5 Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.206881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.206971 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.207054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.207098 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.207282 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:00:54.207572 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.207339 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls podName:02c8db60-a588-40e9-a3b7-ba1ac37a5d47 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.207321614 +0000 UTC m=+38.204092806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mbgph" (UID: "02c8db60-a588-40e9-a3b7-ba1ac37a5d47") : secret "samples-operator-tls" not found Nov 03 20:00:54.208153 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.208127 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.208094203 +0000 UTC m=+38.204865393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : configmap references non-existent config key: service-ca.crt Nov 03 20:00:54.208230 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.208163 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:54.208230 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.208213 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls podName:44590efe-c957-4995-8139-5b54567e51fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.208199887 +0000 UTC m=+38.204971080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-txxdx" (UID: "44590efe-c957-4995-8139-5b54567e51fd") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:54.208230 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.208126 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:00:54.208369 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.208260 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.208249064 +0000 UTC m=+38.205020254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : secret "router-metrics-certs-default" not found Nov 03 20:00:54.263889 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.263867 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll"] Nov 03 20:00:54.266467 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.266440 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb73706_3f89_4a36_83fa_8e5b42d55175.slice/crio-e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102 WatchSource:0}: Error finding container e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102: Status 404 returned error can't find the container with id e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102 Nov 03 20:00:54.308520 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.308357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:54.308620 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308520 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:54.308620 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308536 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68844ff664-xnjc5: secret "image-registry-tls" not found Nov 03 20:00:54.308620 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.308565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:54.308620 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308594 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls podName:2f162f2a-0190-494e-9d6b-0ce6eefa90b5 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.308566773 +0000 UTC m=+38.305337966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls") pod "image-registry-68844ff664-xnjc5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5") : secret "image-registry-tls" not found Nov 03 20:00:54.308742 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.308632 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:54.308742 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308649 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:00:54.308742 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308686 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert podName:4faac008-58e3-41b1-9f56-8a8760071532 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.308675209 +0000 UTC m=+38.305446402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zwgdc" (UID: "4faac008-58e3-41b1-9f56-8a8760071532") : secret "networking-console-plugin-cert" not found Nov 03 20:00:54.308742 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308726 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:54.308742 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308736 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d87675b69-6m79x: secret "image-registry-tls" not found Nov 03 20:00:54.308877 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.308772 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls podName:b843f245-431e-40af-a566-a4fa62de8c39 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.308762286 +0000 UTC m=+38.305533477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls") pod "image-registry-5d87675b69-6m79x" (UID: "b843f245-431e-40af-a566-a4fa62de8c39") : secret "image-registry-tls" not found Nov 03 20:00:54.311964 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.311944 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mt748"] Nov 03 20:00:54.314830 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:54.314804 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed58f76_099a_498d_844d_0069904b507b.slice/crio-1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3 WatchSource:0}: Error finding container 1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3: Status 404 returned error can't find the container with id 1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3 Nov 03 20:00:54.410130 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.410061 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:54.410206 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.410194 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:00:54.410254 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:54.410245 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls podName:99e197f7-de69-4fb1-b6d8-e1ffaa6d4903 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:55.410230682 +0000 UTC m=+38.407001858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls") pod "dns-default-d6zdd" (UID: "99e197f7-de69-4fb1-b6d8-e1ffaa6d4903") : secret "dns-default-metrics-tls" not found Nov 03 20:00:54.581449 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.581426 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:00:54.581565 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.581522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:00:54.581661 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.581641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:00:54.587672 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.587653 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:00:54.587765 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.587681 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\"" Nov 03 20:00:54.587886 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.587866 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:00:54.587886 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.587875 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\"" Nov 03 20:00:54.588043 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.587970 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\"" Nov 03 20:00:54.783394 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.783354 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" event={"ID":"8370c5cf-f442-4e7d-b8ab-ea6ede18552b","Type":"ContainerStarted","Data":"3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9"} Nov 03 20:00:54.784882 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.784851 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" event={"ID":"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e","Type":"ContainerStarted","Data":"deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb"} Nov 03 20:00:54.787184 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.787158 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mt748" event={"ID":"0ed58f76-099a-498d-844d-0069904b507b","Type":"ContainerStarted","Data":"1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3"} Nov 03 20:00:54.788921 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.788897 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" event={"ID":"844916e1-ce51-4692-907d-f30d1b6a9eb8","Type":"ContainerStarted","Data":"f48c3d33d2d6753172bbe829318e73444291aac85c376f7509a1cb469818e025"} Nov 03 20:00:54.789015 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.788939 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" event={"ID":"844916e1-ce51-4692-907d-f30d1b6a9eb8","Type":"ContainerStarted","Data":"c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca"} Nov 03 20:00:54.790091 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.790067 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" event={"ID":"60d3aaa0-ccf0-42ae-92d5-44512e33aed4","Type":"ContainerStarted","Data":"e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5"} Nov 03 20:00:54.791179 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.791163 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" event={"ID":"333ef57f-4887-4c8f-81c8-962265ad3be0","Type":"ContainerStarted","Data":"ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf"} Nov 03 20:00:54.792579 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.792556 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6kg75" event={"ID":"1551a4b4-7bd1-47ec-ae80-e79f4262761e","Type":"ContainerStarted","Data":"9b8e6bb514158914cbef6aca5a0b7c2b737a5e582d2050e452aa01359ace6f4e"} Nov 03 20:00:54.792658 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.792593 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6kg75" event={"ID":"1551a4b4-7bd1-47ec-ae80-e79f4262761e","Type":"ContainerStarted","Data":"f2fadfa5580a834de049f331cb52f046e4c0113af41da9df254671a010bbc1ac"} Nov 03 20:00:54.794018 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.793999 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" event={"ID":"148af517-1bbd-404e-97da-cadd491b1ded","Type":"ContainerStarted","Data":"00320dd5cc5006930ee3c7e45cb1c0afbfce032fceeecd791d7f63b0ad591f33"} Nov 03 20:00:54.794105 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.794035 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" event={"ID":"148af517-1bbd-404e-97da-cadd491b1ded","Type":"ContainerStarted","Data":"c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7"} Nov 03 20:00:54.795287 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.795258 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" event={"ID":"9cb73706-3f89-4a36-83fa-8e5b42d55175","Type":"ContainerStarted","Data":"e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102"} Nov 03 20:00:54.796409 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.796392 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hft4t" event={"ID":"f07a2feb-3648-4f19-9385-baeb5eeeaeeb","Type":"ContainerStarted","Data":"e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3"} Nov 03 20:00:54.808221 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.808183 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" podStartSLOduration=396.808173434 podStartE2EDuration="6m36.808173434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:54.80701708 +0000 UTC m=+37.803788279" watchObservedRunningTime="2026-11-03 20:00:54.808173434 +0000 UTC m=+37.804944631" Nov 03 20:00:54.827640 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.827599 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6kg75" podStartSLOduration=396.827588137 podStartE2EDuration="6m36.827588137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:54.827399784 +0000 UTC m=+37.824170982" watchObservedRunningTime="2026-11-03 20:00:54.827588137 +0000 UTC m=+37.824359335" Nov 03 20:00:54.848478 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:54.848433 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" podStartSLOduration=396.848418279 podStartE2EDuration="6m36.848418279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:54.846868855 +0000 UTC m=+37.843640055" watchObservedRunningTime="2026-11-03 20:00:54.848418279 +0000 UTC m=+37.845189478" Nov 03 20:00:55.217038 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.217001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:55.217244 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.217083 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:55.217244 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.217162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:55.217356 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.217250 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:55.218613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.217532 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.217490582 +0000 UTC m=+40.214261771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : configmap references non-existent config key: service-ca.crt Nov 03 20:00:55.218613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.217720 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:55.218613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.217795 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls podName:44590efe-c957-4995-8139-5b54567e51fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.217776493 +0000 UTC m=+40.214547683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-txxdx" (UID: "44590efe-c957-4995-8139-5b54567e51fd") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:55.218613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.217908 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:00:55.218613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.217975 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls podName:02c8db60-a588-40e9-a3b7-ba1ac37a5d47 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.217964168 +0000 UTC m=+40.214735358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mbgph" (UID: "02c8db60-a588-40e9-a3b7-ba1ac37a5d47") : secret "samples-operator-tls" not found Nov 03 20:00:55.219298 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.219279 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:00:55.219372 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.219345 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.219329518 +0000 UTC m=+40.216100712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : secret "router-metrics-certs-default" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.317921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.317955 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.318082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318204 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318218 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68844ff664-xnjc5: secret "image-registry-tls" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318271 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls podName:2f162f2a-0190-494e-9d6b-0ce6eefa90b5 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.318253946 +0000 UTC m=+40.315025138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls") pod "image-registry-68844ff664-xnjc5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5") : secret "image-registry-tls" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318641 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318691 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert podName:4faac008-58e3-41b1-9f56-8a8760071532 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.318677142 +0000 UTC m=+40.315448333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zwgdc" (UID: "4faac008-58e3-41b1-9f56-8a8760071532") : secret "networking-console-plugin-cert" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318748 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318757 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d87675b69-6m79x: secret "image-registry-tls" not found Nov 03 20:00:55.318822 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.318788 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls podName:b843f245-431e-40af-a566-a4fa62de8c39 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.318777534 +0000 UTC m=+40.315548725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls") pod "image-registry-5d87675b69-6m79x" (UID: "b843f245-431e-40af-a566-a4fa62de8c39") : secret "image-registry-tls" not found Nov 03 20:00:55.418994 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:55.418949 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:55.419253 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.419225 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:00:55.419369 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:55.419291 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls podName:99e197f7-de69-4fb1-b6d8-e1ffaa6d4903 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.41927175 +0000 UTC m=+40.416042935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls") pod "dns-default-d6zdd" (UID: "99e197f7-de69-4fb1-b6d8-e1ffaa6d4903") : secret "dns-default-metrics-tls" not found Nov 03 20:00:56.805987 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:56.805436 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" event={"ID":"60d3aaa0-ccf0-42ae-92d5-44512e33aed4","Type":"ContainerStarted","Data":"b20f19518f65654113705258589f93deb28bfd65c355437e2769cc692400380f"} Nov 03 20:00:56.823888 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:56.823844 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" podStartSLOduration=355.837553024 podStartE2EDuration="5m57.823832137s" totalImagesPullingTime="1.986279113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:56.823572313 +0000 UTC m=+39.820343512" watchObservedRunningTime="2026-11-03 20:00:56.823832137 +0000 UTC m=+39.820603335" Nov 03 20:00:57.238855 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.238811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:57.239035 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.238915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:00:57.239035 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.238962 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:00:57.239119 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.239032 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:00:57.239217 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239200 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:00:57.239282 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239271 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.23925337 +0000 UTC m=+44.236024569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : secret "router-metrics-certs-default" not found Nov 03 20:00:57.239710 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239686 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.239666471 +0000 UTC m=+44.236437662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : configmap references non-existent config key: service-ca.crt Nov 03 20:00:57.239809 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239706 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:00:57.239809 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239730 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:57.239809 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239771 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls podName:02c8db60-a588-40e9-a3b7-ba1ac37a5d47 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.239752474 +0000 UTC m=+44.236523668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mbgph" (UID: "02c8db60-a588-40e9-a3b7-ba1ac37a5d47") : secret "samples-operator-tls" not found Nov 03 20:00:57.239809 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.239787 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls podName:44590efe-c957-4995-8139-5b54567e51fd nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.239780568 +0000 UTC m=+44.236551743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-txxdx" (UID: "44590efe-c957-4995-8139-5b54567e51fd") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.339649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.339811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.339849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.339981 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340036 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert podName:4faac008-58e3-41b1-9f56-8a8760071532 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.340020102 +0000 UTC m=+44.336791283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zwgdc" (UID: "4faac008-58e3-41b1-9f56-8a8760071532") : secret "networking-console-plugin-cert" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340100 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340110 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d87675b69-6m79x: secret "image-registry-tls" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340142 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls podName:b843f245-431e-40af-a566-a4fa62de8c39 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.340132921 +0000 UTC m=+44.336904100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls") pod "image-registry-5d87675b69-6m79x" (UID: "b843f245-431e-40af-a566-a4fa62de8c39") : secret "image-registry-tls" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340195 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340206 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68844ff664-xnjc5: secret "image-registry-tls" not found Nov 03 20:00:57.340538 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.340237 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls podName:2f162f2a-0190-494e-9d6b-0ce6eefa90b5 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.340226827 +0000 UTC m=+44.336998017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls") pod "image-registry-68844ff664-xnjc5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5") : secret "image-registry-tls" not found Nov 03 20:00:57.441243 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:57.441208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:00:57.441553 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.441534 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:00:57.441643 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:57.441595 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls podName:99e197f7-de69-4fb1-b6d8-e1ffaa6d4903 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.441578398 +0000 UTC m=+44.438349593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls") pod "dns-default-d6zdd" (UID: "99e197f7-de69-4fb1-b6d8-e1ffaa6d4903") : secret "dns-default-metrics-tls" not found Nov 03 20:00:58.816556 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:58.816337 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hft4t" event={"ID":"f07a2feb-3648-4f19-9385-baeb5eeeaeeb","Type":"ContainerStarted","Data":"27d9102d81a8eae6b6102c3c76ea548adcea5725476d04366079f88e31103ab3"} Nov 03 20:00:59.016386 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.016323 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-hft4t" podStartSLOduration=356.930040466 podStartE2EDuration="6m1.0163014s" totalImagesPullingTime="4.086260934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:58.84023169 +0000 UTC m=+41.837002888" watchObservedRunningTime="2026-11-03 20:00:59.0163014 +0000 UTC m=+42.013072599" Nov 03 20:00:59.017689 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.017668 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9p8gl"] Nov 03 20:00:59.042761 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.042690 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9p8gl"] Nov 03 20:00:59.042761 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.042713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.042761 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.042754 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-g9msr-f58ll"] Nov 03 20:00:59.049380 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.049301 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:00:59.049495 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.049408 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mszh\"" Nov 03 20:00:59.049495 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.049461 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:00:59.058403 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.058282 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g9msr-f58ll"] Nov 03 20:00:59.058403 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.058379 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.159113 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.159286 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159130 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649bz\" (UniqueName: \"kubernetes.io/projected/17053989-3da4-4c70-92c3-949cd0b733ee-kube-api-access-649bz\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.159286 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.159286 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159255 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17053989-3da4-4c70-92c3-949cd0b733ee-crio-socket\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.159474 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17053989-3da4-4c70-92c3-949cd0b733ee-data-volume\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.159474 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159324 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.159474 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159352 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.159474 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.159691 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.159585 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.260420 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260386 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17053989-3da4-4c70-92c3-949cd0b733ee-crio-socket\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.260605 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17053989-3da4-4c70-92c3-949cd0b733ee-data-volume\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.260605 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.260605 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.260605 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.260812 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260622 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17053989-3da4-4c70-92c3-949cd0b733ee-crio-socket\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.260812 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.260812 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.260812 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260790 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649bz\" (UniqueName: \"kubernetes.io/projected/17053989-3da4-4c70-92c3-949cd0b733ee-kube-api-access-649bz\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.261018 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.261018 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.260851 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17053989-3da4-4c70-92c3-949cd0b733ee-data-volume\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.261018 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.261011 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.261335 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.261314 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:00:59.261440 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.261384 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:00:59.761367159 +0000 UTC m=+42.758138362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : configmap "kube-rbac-proxy" not found Nov 03 20:00:59.261533 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.261443 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:00:59.261533 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.261489 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:00:59.761478113 +0000 UTC m=+42.758249288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : secret "insights-runtime-extractor-tls" not found Nov 03 20:00:59.261652 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.261545 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.261993 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.261973 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.277906 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.277875 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.283130 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.283106 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-649bz\" (UniqueName: \"kubernetes.io/projected/17053989-3da4-4c70-92c3-949cd0b733ee-kube-api-access-649bz\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.419864 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.419794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:00:59.731135 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.730670 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t"] Nov 03 20:00:59.754008 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.753973 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t"] Nov 03 20:00:59.754169 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.754104 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:00:59.755575 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.755550 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj"] Nov 03 20:00:59.766865 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.766840 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:00:59.768100 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.768065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.768186 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.768107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:00:59.768231 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.768205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xqn\" (UniqueName: \"kubernetes.io/projected/8ba183e8-f0a3-41fa-9247-20576448a07f-kube-api-access-r5xqn\") pod \"migrator-66df5d496b-xmd4t\" (UID: \"8ba183e8-f0a3-41fa-9247-20576448a07f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:00:59.768601 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.768577 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:00:59.768678 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.768637 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:01:00.768619027 +0000 UTC m=+43.765390225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : secret "insights-runtime-extractor-tls" not found Nov 03 20:00:59.768990 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.768968 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:00:59.769036 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:00:59.769028 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:01:00.769013058 +0000 UTC m=+43.765784251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : configmap "kube-rbac-proxy" not found Nov 03 20:00:59.770534 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.770087 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-g9msr-f58ll"] Nov 03 20:00:59.770534 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.770202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:00:59.771896 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.771771 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:59.772247 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:00:59.772195 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033c2d08_c440_495d_9957_0b7d41914e1a.slice/crio-b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff WatchSource:0}: Error finding container b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff: Status 404 returned error can't find the container with id b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff Nov 03 20:00:59.772247 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.772242 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n4r7h\"" Nov 03 20:00:59.773642 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.772898 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj"] Nov 03 20:00:59.828815 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.828291 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" event={"ID":"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e","Type":"ContainerStarted","Data":"9799885ea4c0085789344f3e6493f558cf3321afaf5af14f18ecfc114ee922a8"} Nov 03 20:00:59.832598 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.830084 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mt748" event={"ID":"0ed58f76-099a-498d-844d-0069904b507b","Type":"ContainerStarted","Data":"0f143f45a36312a5ad5d4fa60302ec1a96cd1709ebd4b04fa49760456f3561b6"} Nov 03 20:00:59.835139 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.835112 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9msr-f58ll" event={"ID":"033c2d08-c440-495d-9957-0b7d41914e1a","Type":"ContainerStarted","Data":"b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff"} Nov 03 20:00:59.869177 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.869151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xqn\" (UniqueName: \"kubernetes.io/projected/8ba183e8-f0a3-41fa-9247-20576448a07f-kube-api-access-r5xqn\") pod \"migrator-66df5d496b-xmd4t\" (UID: \"8ba183e8-f0a3-41fa-9247-20576448a07f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:00:59.875958 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.875912 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" podStartSLOduration=357.228926687 podStartE2EDuration="6m1.875895028s" totalImagesPullingTime="4.646968341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:59.851778292 +0000 UTC m=+42.848549493" watchObservedRunningTime="2026-11-03 20:00:59.875895028 +0000 UTC m=+42.872666231" Nov 03 20:00:59.877086 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.877051 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mt748" podStartSLOduration=2.412610551 podStartE2EDuration="6.877035965s" totalImagesPullingTime="4.464425414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:59.874432192 +0000 UTC m=+42.871203394" watchObservedRunningTime="2026-11-03 20:00:59.877035965 +0000 UTC m=+42.873807164" Nov 03 20:00:59.885175 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.885154 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xqn\" (UniqueName: \"kubernetes.io/projected/8ba183e8-f0a3-41fa-9247-20576448a07f-kube-api-access-r5xqn\") pod \"migrator-66df5d496b-xmd4t\" (UID: \"8ba183e8-f0a3-41fa-9247-20576448a07f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:00:59.971552 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:00:59.971028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fwf9\" (UniqueName: \"kubernetes.io/projected/24c63ae0-ce91-4f48-9690-45033ad525cb-kube-api-access-5fwf9\") pod \"migrator-66df5d496b-qtrbj\" (UID: \"24c63ae0-ce91-4f48-9690-45033ad525cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:01:00.072303 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.072218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwf9\" (UniqueName: \"kubernetes.io/projected/24c63ae0-ce91-4f48-9690-45033ad525cb-kube-api-access-5fwf9\") pod \"migrator-66df5d496b-qtrbj\" (UID: \"24c63ae0-ce91-4f48-9690-45033ad525cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:01:00.086175 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.086111 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwf9\" (UniqueName: \"kubernetes.io/projected/24c63ae0-ce91-4f48-9690-45033ad525cb-kube-api-access-5fwf9\") pod \"migrator-66df5d496b-qtrbj\" (UID: \"24c63ae0-ce91-4f48-9690-45033ad525cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:01:00.092927 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.092900 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:01:00.128641 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.128618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:01:00.391018 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.390904 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t"] Nov 03 20:01:00.393338 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:00.393296 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba183e8_f0a3_41fa_9247_20576448a07f.slice/crio-57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438 WatchSource:0}: Error finding container 57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438: Status 404 returned error can't find the container with id 57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438 Nov 03 20:01:00.444317 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.444280 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj"] Nov 03 20:01:00.448931 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:00.448907 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c63ae0_ce91_4f48_9690_45033ad525cb.slice/crio-c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e WatchSource:0}: Error finding container c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e: Status 404 returned error can't find the container with id c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e Nov 03 20:01:00.777814 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.777782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:00.777947 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.777818 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:00.777947 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:00.777915 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:01:00.778034 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:00.777966 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.777953394 +0000 UTC m=+45.774724587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : secret "insights-runtime-extractor-tls" not found Nov 03 20:01:00.778242 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.778225 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:00.838847 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.838817 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" event={"ID":"24c63ae0-ce91-4f48-9690-45033ad525cb","Type":"ContainerStarted","Data":"c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e"} Nov 03 20:01:00.840272 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.840249 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9msr-f58ll" event={"ID":"033c2d08-c440-495d-9957-0b7d41914e1a","Type":"ContainerStarted","Data":"65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a"} Nov 03 20:01:00.841569 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.841542 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" event={"ID":"8ba183e8-f0a3-41fa-9247-20576448a07f","Type":"ContainerStarted","Data":"57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438"} Nov 03 20:01:00.860460 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:00.860424 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-g9msr-f58ll" podStartSLOduration=1.860413076 podStartE2EDuration="1.860413076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:00.86036841 +0000 UTC m=+43.857139608" watchObservedRunningTime="2026-11-03 20:01:00.860413076 +0000 UTC m=+43.857184273" Nov 03 20:01:01.283490 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.283264 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:01.283632 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283428 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:01:01.283632 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.283571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:01.283632 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283607 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls podName:44590efe-c957-4995-8139-5b54567e51fd nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.283588346 +0000 UTC m=+52.280359549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-txxdx" (UID: "44590efe-c957-4995-8139-5b54567e51fd") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:01:01.283791 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.283687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:01.283791 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283774 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:01:01.283884 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.283794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:01.283884 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283813 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:01:01.283884 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283825 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls podName:02c8db60-a588-40e9-a3b7-ba1ac37a5d47 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.283810366 +0000 UTC m=+52.280581572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-mbgph" (UID: "02c8db60-a588-40e9-a3b7-ba1ac37a5d47") : secret "samples-operator-tls" not found Nov 03 20:01:01.283884 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283854 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.283843556 +0000 UTC m=+52.280614767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : secret "router-metrics-certs-default" not found Nov 03 20:01:01.284076 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.283913 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.283905576 +0000 UTC m=+52.280676753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : configmap references non-existent config key: service-ca.crt Nov 03 20:01:01.384391 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.384327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:01:01.384391 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.384382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:01.384613 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.384416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:01.384613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384434 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:01:01.384613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384452 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68844ff664-xnjc5: secret "image-registry-tls" not found Nov 03 20:01:01.384613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384500 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls podName:2f162f2a-0190-494e-9d6b-0ce6eefa90b5 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.384487883 +0000 UTC m=+52.381259091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls") pod "image-registry-68844ff664-xnjc5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5") : secret "image-registry-tls" not found Nov 03 20:01:01.384613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384558 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:01:01.384613 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384616 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert podName:4faac008-58e3-41b1-9f56-8a8760071532 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.384599955 +0000 UTC m=+52.381371148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-zwgdc" (UID: "4faac008-58e3-41b1-9f56-8a8760071532") : secret "networking-console-plugin-cert" not found Nov 03 20:01:01.384896 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384636 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:01:01.384896 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384647 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d87675b69-6m79x: secret "image-registry-tls" not found Nov 03 20:01:01.384896 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.384684 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls podName:b843f245-431e-40af-a566-a4fa62de8c39 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.384673456 +0000 UTC m=+52.381444669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls") pod "image-registry-5d87675b69-6m79x" (UID: "b843f245-431e-40af-a566-a4fa62de8c39") : secret "image-registry-tls" not found Nov 03 20:01:01.487595 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.487567 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:01.487750 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.487704 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:01:01.487813 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:01.487761 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls podName:99e197f7-de69-4fb1-b6d8-e1ffaa6d4903 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:09.487744913 +0000 UTC m=+52.484516109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls") pod "dns-default-d6zdd" (UID: "99e197f7-de69-4fb1-b6d8-e1ffaa6d4903") : secret "dns-default-metrics-tls" not found Nov 03 20:01:01.849802 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.849765 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/0.log" Nov 03 20:01:01.850780 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.849815 2572 generic.go:356] "Generic (PLEG): container finished" podID="333ef57f-4887-4c8f-81c8-962265ad3be0" containerID="9a6a691d8f89effdbcd482398b356a22f0348a7131ef2181950721ed4535d7b2" exitCode=255 Nov 03 20:01:01.850780 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.849913 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" event={"ID":"333ef57f-4887-4c8f-81c8-962265ad3be0","Type":"ContainerDied","Data":"9a6a691d8f89effdbcd482398b356a22f0348a7131ef2181950721ed4535d7b2"} Nov 03 20:01:01.850780 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.850184 2572 scope.go:122] "RemoveContainer" containerID="9a6a691d8f89effdbcd482398b356a22f0348a7131ef2181950721ed4535d7b2" Nov 03 20:01:01.852657 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.852412 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" event={"ID":"9cb73706-3f89-4a36-83fa-8e5b42d55175","Type":"ContainerStarted","Data":"6dc10c69e8650a67e24b6e123e54876dfd3f44bf93886c18fdf6923860e89470"} Nov 03 20:01:01.855039 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.855017 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" event={"ID":"8370c5cf-f442-4e7d-b8ab-ea6ede18552b","Type":"ContainerStarted","Data":"de5c36f1e723eeda1251f6019b60209c45a2b0df8c9fbc63b998dcbb6fd29305"} Nov 03 20:01:01.893805 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.893760 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" podStartSLOduration=356.965758131 podStartE2EDuration="6m3.893744189s" totalImagesPullingTime="6.927986058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:01.89312892 +0000 UTC m=+44.889900120" watchObservedRunningTime="2026-11-03 20:01:01.893744189 +0000 UTC m=+44.890515386" Nov 03 20:01:01.919974 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:01.919927 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" podStartSLOduration=357.083962573 podStartE2EDuration="6m3.919913929s" totalImagesPullingTime="6.835951356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:01.918807322 +0000 UTC m=+44.915578522" watchObservedRunningTime="2026-11-03 20:01:01.919913929 +0000 UTC m=+44.916685127" Nov 03 20:01:02.138770 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.138382 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j4p9c_82de9774-f285-46a2-a1d0-cf66b6f780b2/dns-node-resolver/0.log" Nov 03 20:01:02.803445 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.803366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:02.803616 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:02.803531 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:01:02.803616 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:02.803598 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:01:06.803580618 +0000 UTC m=+49.800351807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : secret "insights-runtime-extractor-tls" not found Nov 03 20:01:02.859630 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.859604 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" event={"ID":"8ba183e8-f0a3-41fa-9247-20576448a07f","Type":"ContainerStarted","Data":"a912c2ac1d5e487382c870df5c3629ff64f5faa6cf39a9cda7d4205edc373110"} Nov 03 20:01:02.861148 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.861127 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" event={"ID":"24c63ae0-ce91-4f48-9690-45033ad525cb","Type":"ContainerStarted","Data":"93e7896f1f9339fbc977aa6f2a292d1b76a6d7a362ec4b1147b1a687381ce29b"} Nov 03 20:01:02.862698 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.862681 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/1.log" Nov 03 20:01:02.863060 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.863047 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/0.log" Nov 03 20:01:02.863117 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.863077 2572 generic.go:356] "Generic (PLEG): container finished" podID="333ef57f-4887-4c8f-81c8-962265ad3be0" containerID="213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091" exitCode=255 Nov 03 20:01:02.863117 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.863102 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" event={"ID":"333ef57f-4887-4c8f-81c8-962265ad3be0","Type":"ContainerDied","Data":"213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091"} Nov 03 20:01:02.863181 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.863152 2572 scope.go:122] "RemoveContainer" containerID="9a6a691d8f89effdbcd482398b356a22f0348a7131ef2181950721ed4535d7b2" Nov 03 20:01:02.863461 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.863445 2572 scope.go:122] "RemoveContainer" containerID="213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091" Nov 03 20:01:02.863683 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:02.863652 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mxp5k_openshift-console-operator(333ef57f-4887-4c8f-81c8-962265ad3be0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" podUID="333ef57f-4887-4c8f-81c8-962265ad3be0" Nov 03 20:01:02.880303 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.880265 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" podStartSLOduration=1.999338484 podStartE2EDuration="3.880252106s" totalImagesPullingTime="1.880913622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:02.879471979 +0000 UTC m=+45.876243178" watchObservedRunningTime="2026-11-03 20:01:02.880252106 +0000 UTC m=+45.877023336" Nov 03 20:01:02.916735 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:02.916691 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" podStartSLOduration=1.8719156030000002 podStartE2EDuration="3.916679814s" totalImagesPullingTime="2.044764211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:02.915656429 +0000 UTC m=+45.912427626" watchObservedRunningTime="2026-11-03 20:01:02.916679814 +0000 UTC m=+45.913451011" Nov 03 20:01:03.137738 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.137681 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n27x8_75f38eb4-b4e8-4673-a540-24c8ebe84ee8/node-ca/0.log" Nov 03 20:01:03.737227 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.737199 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mt748_0ed58f76-099a-498d-844d-0069904b507b/serve-healthcheck-canary/0.log" Nov 03 20:01:03.861656 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.861631 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:03.861983 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.861673 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:03.868152 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.868132 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/1.log" Nov 03 20:01:03.868438 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.868423 2572 scope.go:122] "RemoveContainer" containerID="213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091" Nov 03 20:01:03.868657 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:03.868636 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mxp5k_openshift-console-operator(333ef57f-4887-4c8f-81c8-962265ad3be0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" podUID="333ef57f-4887-4c8f-81c8-962265ad3be0" Nov 03 20:01:03.937611 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:03.937587 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-qtrbj_24c63ae0-ce91-4f48-9690-45033ad525cb/migrator/0.log" Nov 03 20:01:04.138028 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:04.137961 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-xmd4t_8ba183e8-f0a3-41fa-9247-20576448a07f/migrator/0.log" Nov 03 20:01:04.339219 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:04.339193 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2_714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/kube-storage-version-migrator-operator/0.log" Nov 03 20:01:04.872416 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:04.872391 2572 scope.go:122] "RemoveContainer" containerID="213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091" Nov 03 20:01:04.872779 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:04.872574 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mxp5k_openshift-console-operator(333ef57f-4887-4c8f-81c8-962265ad3be0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" podUID="333ef57f-4887-4c8f-81c8-962265ad3be0" Nov 03 20:01:06.835810 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:06.835776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:06.836146 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:06.835949 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:01:06.836146 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:06.836019 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls podName:17053989-3da4-4c70-92c3-949cd0b733ee nodeName:}" failed. No retries permitted until 2026-11-03 20:01:14.836003363 +0000 UTC m=+57.832774546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9p8gl" (UID: "17053989-3da4-4c70-92c3-949cd0b733ee") : secret "insights-runtime-extractor-tls" not found Nov 03 20:01:09.354798 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.354767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.354826 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.354858 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.354968 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.355003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.355036 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls podName:44590efe-c957-4995-8139-5b54567e51fd nodeName:}" failed. No retries permitted until 2026-11-03 20:01:25.355022169 +0000 UTC m=+68.351793358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-txxdx" (UID: "44590efe-c957-4995-8139-5b54567e51fd") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.355087 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:01:09.355151 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.355154 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs podName:837adf32-c68a-46ef-bca6-5b5364f74576 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:25.355138325 +0000 UTC m=+68.351909516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs") pod "router-default-585db4bb4c-xs84v" (UID: "837adf32-c68a-46ef-bca6-5b5364f74576") : secret "router-metrics-certs-default" not found Nov 03 20:01:09.355450 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.355433 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:09.358695 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.358673 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:09.423875 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.423855 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:09.456407 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.456375 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") pod \"image-registry-68844ff664-xnjc5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:01:09.456546 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.456430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:09.456546 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.456468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:09.456546 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.456516 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:01:09.456546 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.456533 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68844ff664-xnjc5: secret "image-registry-tls" not found Nov 03 20:01:09.456757 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.456584 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls podName:2f162f2a-0190-494e-9d6b-0ce6eefa90b5 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:25.456570546 +0000 UTC m=+68.453341746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls") pod "image-registry-68844ff664-xnjc5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5") : secret "image-registry-tls" not found Nov 03 20:01:09.456757 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.456643 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:01:09.456757 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.456663 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d87675b69-6m79x: secret "image-registry-tls" not found Nov 03 20:01:09.456757 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.456736 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls podName:b843f245-431e-40af-a566-a4fa62de8c39 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:25.456716689 +0000 UTC m=+68.453487887 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls") pod "image-registry-5d87675b69-6m79x" (UID: "b843f245-431e-40af-a566-a4fa62de8c39") : secret "image-registry-tls" not found Nov 03 20:01:09.458937 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.458917 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:09.547405 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.547366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:09.557944 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.557910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:09.558104 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.558082 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:01:09.558224 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:09.558158 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls podName:99e197f7-de69-4fb1-b6d8-e1ffaa6d4903 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:25.558137507 +0000 UTC m=+68.554908686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls") pod "dns-default-d6zdd" (UID: "99e197f7-de69-4fb1-b6d8-e1ffaa6d4903") : secret "dns-default-metrics-tls" not found Nov 03 20:01:09.696084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.696059 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph"] Nov 03 20:01:09.816658 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.816635 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-zwgdc"] Nov 03 20:01:09.818764 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:09.818732 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4faac008_58e3_41b1_9f56_8a8760071532.slice/crio-4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9 WatchSource:0}: Error finding container 4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9: Status 404 returned error can't find the container with id 4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9 Nov 03 20:01:09.888263 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.888232 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" event={"ID":"02c8db60-a588-40e9-a3b7-ba1ac37a5d47","Type":"ContainerStarted","Data":"81f33476d9ea83fc71ae2dde2ad1362efb651e4b12082c9d774c2237e73a3263"} Nov 03 20:01:09.889344 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:09.889327 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" event={"ID":"4faac008-58e3-41b1-9f56-8a8760071532","Type":"ContainerStarted","Data":"4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9"} Nov 03 20:01:11.898314 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:11.898280 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" event={"ID":"4faac008-58e3-41b1-9f56-8a8760071532","Type":"ContainerStarted","Data":"412c9195c3ffe958c14731a677405885773000463bdf1ec28ce2040e422782bb"} Nov 03 20:01:11.900084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:11.900063 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" event={"ID":"02c8db60-a588-40e9-a3b7-ba1ac37a5d47","Type":"ContainerStarted","Data":"f80ac06f623e284e90d43d92e02522206de4ea73f7ccfd7466cebc6c86f59014"} Nov 03 20:01:11.900193 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:11.900101 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" event={"ID":"02c8db60-a588-40e9-a3b7-ba1ac37a5d47","Type":"ContainerStarted","Data":"568d454c701cf99ce129d477e331841e090058feff679ca00978aab1c346d721"} Nov 03 20:01:11.917242 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:11.917203 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" podStartSLOduration=346.47409571 podStartE2EDuration="5m47.917190156s" totalImagesPullingTime="1.443094446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:11.915967441 +0000 UTC m=+54.912738651" watchObservedRunningTime="2026-11-03 20:01:11.917190156 +0000 UTC m=+54.913961377" Nov 03 20:01:11.935907 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:11.935871 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" podStartSLOduration=372.297258521 podStartE2EDuration="6m13.935859244s" totalImagesPullingTime="1.638600723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:11.935108753 +0000 UTC m=+54.931879954" watchObservedRunningTime="2026-11-03 20:01:11.935859244 +0000 UTC m=+54.932630441" Nov 03 20:01:13.581818 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.581781 2572 scope.go:122] "RemoveContainer" containerID="213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091" Nov 03 20:01:13.907789 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.907729 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:01:13.908165 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.908150 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/1.log" Nov 03 20:01:13.908209 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.908182 2572 generic.go:356] "Generic (PLEG): container finished" podID="333ef57f-4887-4c8f-81c8-962265ad3be0" containerID="c0952f28baf936512f566d16da0236fd83f8112e3a0beebedfa0c585a9628b48" exitCode=255 Nov 03 20:01:13.908279 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.908260 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" event={"ID":"333ef57f-4887-4c8f-81c8-962265ad3be0","Type":"ContainerDied","Data":"c0952f28baf936512f566d16da0236fd83f8112e3a0beebedfa0c585a9628b48"} Nov 03 20:01:13.908328 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.908318 2572 scope.go:122] "RemoveContainer" containerID="213b4ad5539bc53989639ed7b2930393cf3da79e77ea3c9b7f09344571b9f091" Nov 03 20:01:13.908620 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:13.908592 2572 scope.go:122] "RemoveContainer" containerID="c0952f28baf936512f566d16da0236fd83f8112e3a0beebedfa0c585a9628b48" Nov 03 20:01:13.908805 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:13.908786 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mxp5k_openshift-console-operator(333ef57f-4887-4c8f-81c8-962265ad3be0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" podUID="333ef57f-4887-4c8f-81c8-962265ad3be0" Nov 03 20:01:14.902890 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:14.902857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:14.905163 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:14.905145 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:14.913432 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:14.913404 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:01:15.003782 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:15.003767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:15.275827 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:15.275794 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9p8gl"] Nov 03 20:01:15.278124 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:15.278094 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17053989_3da4_4c70_92c3_949cd0b733ee.slice/crio-836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3 WatchSource:0}: Error finding container 836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3: Status 404 returned error can't find the container with id 836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3 Nov 03 20:01:15.918738 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:15.918702 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p8gl" event={"ID":"17053989-3da4-4c70-92c3-949cd0b733ee","Type":"ContainerStarted","Data":"63a2a61504922ac7ca357a18a8baedabb5733fff628916c03658e8987a8fce06"} Nov 03 20:01:15.919091 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:15.918769 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p8gl" event={"ID":"17053989-3da4-4c70-92c3-949cd0b733ee","Type":"ContainerStarted","Data":"836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3"} Nov 03 20:01:16.925259 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:16.925225 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p8gl" event={"ID":"17053989-3da4-4c70-92c3-949cd0b733ee","Type":"ContainerStarted","Data":"73e2ec351f0f046523a05ed90add09823bcf9c7524271a7364c85b272ebf90d0"} Nov 03 20:01:17.930469 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:17.930400 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9p8gl" event={"ID":"17053989-3da4-4c70-92c3-949cd0b733ee","Type":"ContainerStarted","Data":"e4f576299fd5d42b24499d46ec0415e01beac555b74bc0e5a9effc00d5f7613a"} Nov 03 20:01:17.953183 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:17.953137 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9p8gl" podStartSLOduration=17.824964431 podStartE2EDuration="19.953124609s" totalImagesPullingTime="2.128160178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:17.951753803 +0000 UTC m=+60.948525001" watchObservedRunningTime="2026-11-03 20:01:17.953124609 +0000 UTC m=+60.949895807" Nov 03 20:01:18.779977 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:18.779952 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:21.045350 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:21.045315 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68844ff664-xnjc5"] Nov 03 20:01:21.045883 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:21.045576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-68844ff664-xnjc5" podUID="2f162f2a-0190-494e-9d6b-0ce6eefa90b5" Nov 03 20:01:21.944206 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:21.944170 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:01:21.952682 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:21.952657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:01:22.061692 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061603 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061726 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061757 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061782 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061816 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061833 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062084 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061853 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token\") pod \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\" (UID: \"2f162f2a-0190-494e-9d6b-0ce6eefa90b5\") " Nov 03 20:01:22.062372 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.062129 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:22.062372 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.061976 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:22.062372 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.062196 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:22.062467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.062425 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-trusted-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.062467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.062446 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-certificates\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.062467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.062463 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-ca-trust-extracted\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.064122 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.064094 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:22.064214 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.064152 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:01:22.064574 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.064550 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "kube-api-access-qmmtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:22.064574 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.064563 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token" pod "2f162f2a-0190-494e-9d6b-0ce6eefa90b5" (UID: "2f162f2a-0190-494e-9d6b-0ce6eefa90b5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:22.162916 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.162889 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-installation-pull-secrets\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.162916 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.162913 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-image-registry-private-configuration\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.163067 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.162924 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-bound-sa-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.163067 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.162934 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmtp\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-kube-api-access-qmmtp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:22.947105 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.947075 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68844ff664-xnjc5" Nov 03 20:01:22.986033 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.986002 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68844ff664-xnjc5"] Nov 03 20:01:22.991834 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:22.991811 2572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68844ff664-xnjc5"] Nov 03 20:01:23.069021 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.068991 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2f162f2a-0190-494e-9d6b-0ce6eefa90b5-registry-tls\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:23.270590 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.270549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:23.270727 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.270693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:23.273518 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.273492 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:01:23.273668 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.273657 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:01:23.284002 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.283968 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:23.284103 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.284029 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:23.433641 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.433619 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\"" Nov 03 20:01:23.441629 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.441615 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:23.472985 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.472959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:23.475462 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.475441 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:23.483224 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.483209 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\"" Nov 03 20:01:23.491289 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.491270 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:23.585814 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.585784 2572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f162f2a-0190-494e-9d6b-0ce6eefa90b5" path="/var/lib/kubelet/pods/2f162f2a-0190-494e-9d6b-0ce6eefa90b5/volumes" Nov 03 20:01:23.711330 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.711309 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6k5bk"] Nov 03 20:01:23.712783 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.712765 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\"" Nov 03 20:01:23.713957 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:23.713936 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7ef6fe_524b_4542_b281_27f7fce2deb8.slice/crio-8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1 WatchSource:0}: Error finding container 8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1: Status 404 returned error can't find the container with id 8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1 Nov 03 20:01:23.720228 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.720211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:23.765181 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.765157 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bpqrl"] Nov 03 20:01:23.769191 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:23.769069 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757e39d9_7832_46de_8c68_930c0b80943e.slice/crio-ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c WatchSource:0}: Error finding container ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c: Status 404 returned error can't find the container with id ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c Nov 03 20:01:23.861670 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.861595 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:23.861670 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.861647 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:23.861967 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.861954 2572 scope.go:122] "RemoveContainer" containerID="c0952f28baf936512f566d16da0236fd83f8112e3a0beebedfa0c585a9628b48" Nov 03 20:01:23.862149 ip-10-0-143-81 kubenswrapper[2572]: E1103 20:01:23.862132 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mxp5k_openshift-console-operator(333ef57f-4887-4c8f-81c8-962265ad3be0)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" podUID="333ef57f-4887-4c8f-81c8-962265ad3be0" Nov 03 20:01:23.951687 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.951654 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpqrl" event={"ID":"757e39d9-7832-46de-8c68-930c0b80943e","Type":"ContainerStarted","Data":"ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c"} Nov 03 20:01:23.952940 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.952912 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6k5bk" event={"ID":"3b7ef6fe-524b-4542-b281-27f7fce2deb8","Type":"ContainerStarted","Data":"8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1"} Nov 03 20:01:23.991410 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:23.991387 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kglzd"] Nov 03 20:01:23.993819 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:23.993792 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7978e0e4_9979_42cf_a15b_767a03352919.slice/crio-70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8 WatchSource:0}: Error finding container 70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8: Status 404 returned error can't find the container with id 70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8 Nov 03 20:01:24.958432 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:24.958389 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kglzd" event={"ID":"7978e0e4-9979-42cf-a15b-767a03352919","Type":"ContainerStarted","Data":"42d88370314a5567cf0d90db9823509ef9c6c73a25275e513cafae4acbc6de6c"} Nov 03 20:01:24.958870 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:24.958443 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kglzd" event={"ID":"7978e0e4-9979-42cf-a15b-767a03352919","Type":"ContainerStarted","Data":"70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8"} Nov 03 20:01:24.958870 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:24.958472 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:25.389948 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.389908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:25.390109 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.389981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:25.392940 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.392893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:25.393313 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.393290 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:25.491408 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.491370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:25.494400 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.494374 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:25.577433 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.577402 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f6jr5\"" Nov 03 20:01:25.584876 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.584847 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:25.592778 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.592751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:25.595406 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.595381 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:25.675088 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.675060 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-grxf8\"" Nov 03 20:01:25.683833 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.682314 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:25.729755 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.729730 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pw8jf\"" Nov 03 20:01:25.737304 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.737283 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:25.828986 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.828961 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\"" Nov 03 20:01:25.837743 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.837717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:25.878069 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.877728 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kglzd" podStartSLOduration=67.877713324 podStartE2EDuration="1m7.877713324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:24.978480665 +0000 UTC m=+67.975251863" watchObservedRunningTime="2026-11-03 20:01:25.877713324 +0000 UTC m=+68.874484536" Nov 03 20:01:25.881615 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.879448 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx"] Nov 03 20:01:25.968802 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.968764 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpqrl" event={"ID":"757e39d9-7832-46de-8c68-930c0b80943e","Type":"ContainerStarted","Data":"9ccd6211130459fbca5ae615d814d2f0ccf7212a3ce12ab4ddec94c5c48311fb"} Nov 03 20:01:25.969192 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.968850 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bpqrl" event={"ID":"757e39d9-7832-46de-8c68-930c0b80943e","Type":"ContainerStarted","Data":"e78764a08988f4447fb03e8a47f21eced0740a011cab63f6268d617405a62da3"} Nov 03 20:01:25.970393 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.970362 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" event={"ID":"44590efe-c957-4995-8139-5b54567e51fd","Type":"ContainerStarted","Data":"9f833588834de38052d27149bbababd688b397edbc07c05420742515021f8c8d"} Nov 03 20:01:25.980268 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.980246 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-585db4bb4c-xs84v"] Nov 03 20:01:25.985234 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:25.985212 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837adf32_c68a_46ef_bca6_5b5364f74576.slice/crio-e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3 WatchSource:0}: Error finding container e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3: Status 404 returned error can't find the container with id e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3 Nov 03 20:01:25.987577 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:25.987526 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bpqrl" podStartSLOduration=66.12460124 podStartE2EDuration="1m7.987494586s" totalImagesPullingTime="1.862893346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:25.986635478 +0000 UTC m=+68.983406678" watchObservedRunningTime="2026-11-03 20:01:25.987494586 +0000 UTC m=+68.984265786" Nov 03 20:01:26.038110 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.038086 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d87675b69-6m79x"] Nov 03 20:01:26.040364 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:26.040338 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb843f245_431e_40af_a566_a4fa62de8c39.slice/crio-6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336 WatchSource:0}: Error finding container 6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336: Status 404 returned error can't find the container with id 6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336 Nov 03 20:01:26.123812 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.123787 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d6zdd"] Nov 03 20:01:26.126279 ip-10-0-143-81 kubenswrapper[2572]: W1103 20:01:26.126255 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e197f7_de69_4fb1_b6d8_e1ffaa6d4903.slice/crio-ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d WatchSource:0}: Error finding container ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d: Status 404 returned error can't find the container with id ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d Nov 03 20:01:26.978730 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.978688 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-585db4bb4c-xs84v" event={"ID":"837adf32-c68a-46ef-bca6-5b5364f74576","Type":"ContainerStarted","Data":"36864b68ad757074645ff5db8fc35e24a7a6b1a74fe2ec9f50a058abb82460f6"} Nov 03 20:01:26.979208 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.978754 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-585db4bb4c-xs84v" event={"ID":"837adf32-c68a-46ef-bca6-5b5364f74576","Type":"ContainerStarted","Data":"e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3"} Nov 03 20:01:26.981291 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.981254 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6zdd" event={"ID":"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903","Type":"ContainerStarted","Data":"ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d"} Nov 03 20:01:26.984549 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.984523 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" event={"ID":"b843f245-431e-40af-a566-a4fa62de8c39","Type":"ContainerStarted","Data":"6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0"} Nov 03 20:01:26.984663 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.984556 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" event={"ID":"b843f245-431e-40af-a566-a4fa62de8c39","Type":"ContainerStarted","Data":"6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336"} Nov 03 20:01:26.984663 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:26.984573 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:27.006526 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:27.006457 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-585db4bb4c-xs84v" podStartSLOduration=388.006443395 podStartE2EDuration="6m28.006443395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:27.003777309 +0000 UTC m=+70.000548507" watchObservedRunningTime="2026-11-03 20:01:27.006443395 +0000 UTC m=+70.003214594" Nov 03 20:01:27.682710 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:27.682682 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:27.685524 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:27.685484 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:27.711379 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:27.711334 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" podStartSLOduration=59.711316499 podStartE2EDuration="59.711316499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:27.030777102 +0000 UTC m=+70.027548302" watchObservedRunningTime="2026-11-03 20:01:27.711316499 +0000 UTC m=+70.708087698" Nov 03 20:01:27.988154 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:27.988129 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:27.989581 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:27.989561 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:28.994305 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:28.994265 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" event={"ID":"44590efe-c957-4995-8139-5b54567e51fd","Type":"ContainerStarted","Data":"bb9de72660625029f4625d35a5deec20c98c7eb09868bb2a1d1c83adaa8ba24e"} Nov 03 20:01:28.996623 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:28.996533 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6zdd" event={"ID":"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903","Type":"ContainerStarted","Data":"c82817500d134454ea2442955eefe97f49166a0e47b5b330d66334621ee2a6f2"} Nov 03 20:01:28.996623 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:28.996588 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6zdd" event={"ID":"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903","Type":"ContainerStarted","Data":"b4ab15478a08a1ae553040076ca0724d3c8c3156e520de25f474e97330bbc3ec"} Nov 03 20:01:29.015467 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:29.015421 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" podStartSLOduration=388.977088881 podStartE2EDuration="6m31.015408682s" totalImagesPullingTime="2.038319801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:29.013869175 +0000 UTC m=+72.010640374" watchObservedRunningTime="2026-11-03 20:01:29.015408682 +0000 UTC m=+72.012179880" Nov 03 20:01:29.040438 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:29.040380 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-d6zdd" podStartSLOduration=34.02544896 podStartE2EDuration="36.040362588s" totalImagesPullingTime="2.014913628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:29.038897558 +0000 UTC m=+72.035668757" watchObservedRunningTime="2026-11-03 20:01:29.040362588 +0000 UTC m=+72.037133786" Nov 03 20:01:30.000524 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:30.000471 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:30.380861 ip-10-0-143-81 kubenswrapper[2572]: I1103 20:01:30.380757 2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:01:30.380922 ip-10-0-143-81 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:01:30.400866 ip-10-0-143-81 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:01:30.401032 ip-10-0-143-81 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:01:30.401207 ip-10-0-143-81 systemd[1]: kubelet.service: Consumed 4.446s CPU time. Nov 03 20:01:30.416114 ip-10-0-143-81 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:01:30.498374 ip-10-0-143-81 kubenswrapper[5815]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:01:30.498374 ip-10-0-143-81 kubenswrapper[5815]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:01:30.498374 ip-10-0-143-81 kubenswrapper[5815]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:01:30.498374 ip-10-0-143-81 kubenswrapper[5815]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:01:30.500398 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500328 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.500398 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500399 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500404 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500408 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500410 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500413 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500416 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500418 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500421 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500423 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500426 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500428 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500431 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500433 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500438 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500442 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500445 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.500469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500474 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500479 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500482 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500486 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500489 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500491 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500494 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500497 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500499 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500518 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500522 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500525 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500527 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500530 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500532 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500534 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500537 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500540 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500543 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500546 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.500893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500548 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500550 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500553 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500555 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500558 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500560 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500563 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500566 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500568 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500571 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500574 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500576 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500579 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500582 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500584 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500589 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500593 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500596 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500599 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.501397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500602 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500605 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500607 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500610 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500613 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500615 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500618 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500621 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500624 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500627 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500629 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500632 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500635 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500637 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500640 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500643 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500645 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500648 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500650 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.501902 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500653 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500656 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500658 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500660 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500663 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500666 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500668 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500671 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500673 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500675 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500679 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500681 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500684 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500686 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500689 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500692 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500694 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500696 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500699 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500702 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.502378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500707 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500709 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500712 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500715 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500717 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500720 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500722 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500725 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500727 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500730 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.500732 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501463 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501470 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501473 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501476 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501478 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501481 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501484 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501486 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.502900 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501489 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501491 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501493 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501497 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501500 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501520 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501524 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501528 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501532 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501536 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501539 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501543 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501547 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501553 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501557 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501562 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501567 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501571 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501575 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.503360 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501577 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501580 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501582 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501585 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501588 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501590 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501593 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501595 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501599 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501602 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501604 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501607 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501610 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501612 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501615 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501618 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501620 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501623 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501626 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501628 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.503857 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501631 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501634 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501636 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501638 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501641 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501643 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501647 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501650 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501653 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501655 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501658 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501660 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501663 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501665 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501667 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501670 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501673 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501675 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501678 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.504401 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501680 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501683 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501689 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501692 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501694 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501697 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501699 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501702 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501704 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501708 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501711 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501713 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501716 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501718 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501721 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501723 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501726 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501728 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501730 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501734 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.504893 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501737 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501739 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501741 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501744 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501747 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501749 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501751 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501754 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501756 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501759 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501761 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501764 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501766 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501769 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501771 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501774 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501779 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501783 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501786 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.501788 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.505408 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502258 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502262 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502265 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502267 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502270 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502273 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502277 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502280 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502283 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502286 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502289 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502293 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502296 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502299 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502301 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502304 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502306 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502309 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502312 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502314 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.505944 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502317 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502319 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502322 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502324 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502327 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502329 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502332 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502335 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502338 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502340 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502342 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502345 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502348 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502351 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502353 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502356 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502358 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502361 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502363 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502366 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.506851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502369 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502371 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502374 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502377 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502380 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502382 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502385 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502388 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502390 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502393 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502396 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502398 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502401 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502403 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502406 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502408 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502411 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502413 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502416 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502418 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.507740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502421 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502424 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502426 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502429 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502431 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502433 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502436 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502438 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502441 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502443 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502445 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502448 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502450 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502453 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502455 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502459 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502461 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502464 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502466 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.508619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502468 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502471 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502473 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502476 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502478 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502481 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502483 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502486 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502489 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502493 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502497 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502501 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502524 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502528 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502532 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502535 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502538 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502541 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502544 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.509376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502546 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502549 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502552 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502556 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502558 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502561 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502563 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.502567 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502658 5815 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502667 5815 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502674 5815 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502681 5815 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502685 5815 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502689 5815 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502693 5815 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502697 5815 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502700 5815 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502703 5815 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502706 5815 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502709 5815 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502712 5815 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502715 5815 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502718 5815 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:01:30.509882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502721 5815 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502724 5815 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502726 5815 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502731 5815 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502734 5815 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502738 5815 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502741 5815 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502744 5815 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502748 5815 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502751 5815 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502754 5815 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502757 5815 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502762 5815 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502765 5815 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502768 5815 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502771 5815 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502774 5815 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502778 5815 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502781 5815 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502784 5815 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502787 5815 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502790 5815 flags.go:64] FLAG: --enable-server="true" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502793 5815 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502797 5815 flags.go:64] FLAG: --event-burst="100" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502800 5815 flags.go:64] FLAG: --event-qps="50" Nov 03 20:01:30.510457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502803 5815 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502806 5815 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502809 5815 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502813 5815 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502816 5815 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502819 5815 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502821 5815 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502824 5815 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502827 5815 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502830 5815 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502833 5815 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502836 5815 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502838 5815 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502847 5815 flags.go:64] FLAG: --feature-gates="" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502852 5815 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502855 5815 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502858 5815 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502861 5815 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502863 5815 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502868 5815 flags.go:64] FLAG: --help="false" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502870 5815 flags.go:64] FLAG: --hostname-override="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502874 5815 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502876 5815 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:01:30.511126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502879 5815 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502882 5815 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502887 5815 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502890 5815 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502893 5815 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502896 5815 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502898 5815 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502901 5815 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502905 5815 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502907 5815 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502910 5815 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502913 5815 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502916 5815 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502919 5815 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502921 5815 flags.go:64] FLAG: --lock-file="" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502924 5815 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502927 5815 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502930 5815 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502936 5815 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502938 5815 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502941 5815 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502944 5815 flags.go:64] FLAG: --logging-format="text" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502947 5815 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502956 5815 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:01:30.511715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502959 5815 flags.go:64] FLAG: --manifest-url="" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502962 5815 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502966 5815 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502969 5815 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502974 5815 flags.go:64] FLAG: --max-pods="110" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502977 5815 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502980 5815 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502983 5815 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502986 5815 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502989 5815 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502992 5815 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.502995 5815 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503002 5815 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503005 5815 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503007 5815 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503011 5815 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503013 5815 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503016 5815 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503019 5815 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503022 5815 flags.go:64] FLAG: --port="10250" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503025 5815 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503028 5815 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-012a04d55116578a2" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503031 5815 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503033 5815 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503036 5815 flags.go:64] FLAG: --register-node="true" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503039 5815 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:01:30.512295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503043 5815 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503045 5815 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503048 5815 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503050 5815 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503054 5815 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503057 5815 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503065 5815 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503068 5815 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503071 5815 flags.go:64] FLAG: --runonce="false" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503074 5815 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503078 5815 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503081 5815 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503083 5815 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503086 5815 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503089 5815 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503092 5815 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503096 5815 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503099 5815 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503102 5815 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503105 5815 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503107 5815 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503110 5815 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503113 5815 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503116 5815 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503121 5815 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:01:30.513018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503124 5815 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503126 5815 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503133 5815 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503136 5815 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503138 5815 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503141 5815 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503144 5815 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503147 5815 flags.go:64] FLAG: --v="2" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503151 5815 flags.go:64] FLAG: --version="false" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503155 5815 flags.go:64] FLAG: --vmodule="" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503159 5815 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503162 5815 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503284 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503287 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503290 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503293 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503296 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503300 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503302 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503305 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503309 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503312 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.513722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503315 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503319 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503322 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503324 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503327 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503330 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503332 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503335 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503337 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503340 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503342 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503345 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503348 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503350 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503353 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503355 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503358 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503360 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503363 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503365 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.514377 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503368 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503371 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503373 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503376 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503378 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503381 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503384 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503388 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503390 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503393 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503395 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503397 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503400 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503403 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503406 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503408 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503411 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503413 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503415 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503418 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.515022 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503420 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503423 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503425 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503428 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503430 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503433 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503435 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503438 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503440 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503443 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503445 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503448 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503450 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503453 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503455 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503458 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503460 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503464 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503470 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.515971 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503475 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503478 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503481 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503483 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503485 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503488 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503490 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503493 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503495 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503498 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503500 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503520 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503524 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503528 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503531 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503534 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503536 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503539 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503541 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503544 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.516734 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503546 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503549 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503551 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503553 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503556 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503559 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503561 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503563 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503566 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503568 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503571 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503573 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503585 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503588 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503591 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503593 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.517320 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503595 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.503621 5815 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503654 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503658 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503661 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503664 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503666 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503669 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503672 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503674 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503677 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503679 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503681 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503684 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503686 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503689 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.517811 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503691 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503694 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503696 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503699 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503701 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503704 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503706 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503708 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503711 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503713 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503715 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503718 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503721 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503729 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503732 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503734 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503737 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503739 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503742 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503744 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.518212 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503746 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503750 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503753 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503756 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503758 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503761 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503763 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503766 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503768 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503770 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503773 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503776 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503779 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503781 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503784 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503786 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503789 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503791 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503794 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.518727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503796 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503799 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503801 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503803 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503806 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503810 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503812 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503816 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503818 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503821 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503823 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503825 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503828 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503830 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503833 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503835 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503838 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503841 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503843 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503846 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.519183 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503848 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503850 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503853 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503856 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503861 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503864 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503866 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503869 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503872 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503875 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503879 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503882 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503885 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503888 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503890 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503893 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503896 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503899 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503902 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.519700 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503904 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503910 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503913 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503916 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503919 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503922 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503924 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503927 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503929 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503932 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503934 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503937 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503939 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.503941 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.508299 5815 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.508438 5815 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.508472 5815 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.508480 5815 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508684 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508704 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.520232 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508710 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508715 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508719 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508723 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508728 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508732 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508736 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508740 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508745 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508750 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508754 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508763 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508770 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508776 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508781 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508785 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508789 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508793 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508797 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.520778 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508801 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508805 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508810 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508814 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508818 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508827 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508831 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508835 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508839 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508845 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508849 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508854 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508858 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508862 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508866 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508870 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508875 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508879 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508888 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.521241 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508893 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508897 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508902 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508906 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508910 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508914 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508921 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508927 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508932 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508937 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508941 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508950 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508954 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508958 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508962 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508966 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508970 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508974 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508978 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508982 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.521741 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508986 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508990 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508994 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.508999 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509008 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509012 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509017 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509021 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509025 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509029 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509033 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509037 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509041 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509045 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509049 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509053 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509062 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509067 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509071 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509075 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.522265 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509079 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509083 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509087 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509092 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509096 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509100 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509104 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509108 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509112 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509121 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509125 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509130 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509134 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509138 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509142 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509146 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509151 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509155 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509160 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.522787 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509164 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509168 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509177 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509181 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509185 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509189 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509193 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.509246 5815 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509710 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509723 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509727 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509729 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509732 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509735 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509737 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509740 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.523258 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509743 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509745 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509748 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509751 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509753 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509756 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509759 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509761 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509764 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509766 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509769 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509772 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509776 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509781 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509785 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509788 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509791 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509793 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509796 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.523718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509799 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509801 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509804 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509806 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509809 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509812 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509815 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509818 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509820 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509822 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509825 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509827 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509830 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509833 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509835 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509838 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509840 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509843 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509845 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.524226 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509848 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509851 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509853 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509855 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509858 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509860 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509864 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509868 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509872 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509874 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509877 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509879 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509882 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509885 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509887 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509890 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509893 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509895 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509898 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.524854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509901 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509903 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509906 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509908 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509910 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509916 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509918 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509921 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509924 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509926 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509929 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509932 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509935 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509937 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509940 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509942 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509945 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509947 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509950 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509952 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.525655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509955 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509957 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509960 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509962 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509965 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509968 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509970 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509972 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509975 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509978 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509980 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509982 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509985 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509987 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509990 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509992 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509995 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.509998 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510001 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510004 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.526295 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510008 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.510039 5815 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510075 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510079 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510082 5815 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510085 5815 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510087 5815 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510090 5815 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510092 5815 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510095 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510097 5815 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510100 5815 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510102 5815 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510104 5815 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510107 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510109 5815 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:30.526935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510112 5815 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510114 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510117 5815 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510119 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510122 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510124 5815 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510126 5815 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510129 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510132 5815 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510134 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510136 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510139 5815 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510141 5815 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510145 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510152 5815 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510156 5815 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510159 5815 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510162 5815 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510165 5815 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:30.527332 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510168 5815 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510170 5815 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510173 5815 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510175 5815 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510178 5815 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510180 5815 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510183 5815 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510185 5815 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510188 5815 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510190 5815 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510193 5815 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510195 5815 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510198 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510200 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510203 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510205 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510208 5815 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510210 5815 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510213 5815 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510215 5815 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:30.527935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510218 5815 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510220 5815 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510223 5815 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510225 5815 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510228 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510230 5815 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510234 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510238 5815 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510240 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510243 5815 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510246 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510249 5815 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510252 5815 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510254 5815 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510257 5815 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510259 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510262 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510264 5815 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510267 5815 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510269 5815 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:30.528658 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510272 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510274 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510277 5815 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510279 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510282 5815 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510284 5815 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510286 5815 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510289 5815 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510291 5815 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510294 5815 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510296 5815 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510299 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510301 5815 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510303 5815 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510306 5815 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510308 5815 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510311 5815 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510313 5815 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510317 5815 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510321 5815 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:30.529293 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510323 5815 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510326 5815 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510328 5815 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510331 5815 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510334 5815 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510337 5815 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510339 5815 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510341 5815 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510344 5815 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510348 5815 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510352 5815 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510354 5815 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:30.510357 5815 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.510415 5815 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.511431 5815 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.511484 5815 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.511675 5815 server.go:1014] "Starting client certificate rotation" Nov 03 20:01:30.529821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.511828 5815 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.511872 5815 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:55:17 +0000 UTC" deadline="2028-08-12 18:31:11.62160857 +0000 UTC" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.511897 5815 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15550h29m41.109713642s" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.512088 5815 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.513133 5815 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.514204 5815 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.516561 5815 log.go:25] "Validated CRI v1 image API" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.517162 5815 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:01:30.530269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.519321 5815 fs.go:135] Filesystem UUIDs: map[12621e8d-2a8b-4cc6-bda6-89c9305ff98e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 87584bb3-24e6-4223-90b4-3b26dce39c69:/dev/nvme0n1p3] Nov 03 20:01:30.530594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.519339 5815 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9/userdata/shm major:0 minor:360 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81f33476d9ea83fc71ae2dde2ad1362efb651e4b12082c9d774c2237e73a3263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81f33476d9ea83fc71ae2dde2ad1362efb651e4b12082c9d774c2237e73a3263/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f833588834de38052d27149bbababd688b397edbc07c05420742515021f8c8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f833588834de38052d27149bbababd688b397edbc07c05420742515021f8c8d/userdata/shm major:0 minor:427 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3edb09c604bd7b56aa109dde93c7b54eaa04361d45ac9a599918194cc92b03f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3edb09c604bd7b56aa109dde93c7b54eaa04361d45ac9a599918194cc92b03f/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c/userdata/shm major:0 minor:408 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3/userdata/shm major:0 minor:432 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed35a41a4635db4b29bb696539cc66db4f5aa2947d51d44ba736c2dc44971e0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed35a41a4635db4b29bb696539cc66db4f5aa2947d51d44ba736c2dc44971e0d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f2fadfa5580a834de049f331cb52f046e4c0113af41da9df254671a010bbc1ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f2fadfa5580a834de049f331cb52f046e4c0113af41da9df254671a010bbc1ac/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02c8db60-a588-40e9-a3b7-ba1ac37a5d47/volumes/kubernetes.io~projected/kube-api-access-7kckq:{mountpoint:/var/lib/kubelet/pods/02c8db60-a588-40e9-a3b7-ba1ac37a5d47/volumes/kubernetes.io~projected/kube-api-access-7kckq major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02c8db60-a588-40e9-a3b7-ba1ac37a5d47/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/02c8db60-a588-40e9-a3b7-ba1ac37a5d47/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/033c2d08-c440-495d-9957-0b7d41914e1a/volumes/kubernetes.io~projected/kube-api-access-r7nbn:{mountpoint:/var/lib/kubelet/pods/033c2d08-c440-495d-9957-0b7d41914e1a/volumes/kubernetes.io~projected/kube-api-access-r7nbn major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~projected/kube-api-access-6mvpb:{mountpoint:/var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~projected/kube-api-access-6mvpb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ed58f76-099a-498d-844d-0069904b507b/volumes/kubernetes.io~projected/kube-api-access-qbz8z:{mountpoint:/var/lib/kubelet/pods/0ed58f76-099a-498d-844d-0069904b507b/volumes/kubernetes.io~projected/kube-api-access-qbz8z major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ed58f76-099a-498d-844d-0069904b507b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0ed58f76-099a-498d-844d-0069904b507b/volumes/kubernetes.io~secret/cert major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/148af517-1bbd-404e-97da-cadd491b1ded/volumes/kubernetes.io~projected/kube-api-access-6m7rm:{mountpoint:/var/lib/kubelet/pods/148af517-1bbd-404e-97da-cadd491b1ded/volumes/kubernetes.io~projected/kube-api-access-6m7rm major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1551a4b4-7bd1-47ec-ae80-e79f4262761e/volumes/kubernetes.io~projected/kube-api-access-9r6fx:{mountpoint:/var/lib/kubelet/pods/1551a4b4-7bd1-47ec-ae80-e79f4262761e/volumes/kubernetes.io~projected/kube-api-access-9r6fx major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1679fbb3-a322-4c15-9837-df1040cc3f01/volumes/kubernetes.io~projected/kube-api-access-hgbwh:{mountpoint:/var/lib/kubelet/pods/1679fbb3-a322-4c15-9837-df1040cc3f01/volumes/kubernetes.io~projected/kube-api-access-hgbwh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17053989-3da4-4c70-92c3-949cd0b733ee/volumes/kubernetes.io~projected/kube-api-access-649bz:{mountpoint:/var/lib/kubelet/pods/17053989-3da4-4c70-92c3-949cd0b733ee/volumes/kubernetes.io~projected/kube-api-access-649bz major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17053989-3da4-4c70-92c3-949cd0b733ee/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/17053989-3da4-4c70-92c3-949cd0b733ee/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c63ae0-ce91-4f48-9690-45033ad525cb/volumes/kubernetes.io~projected/kube-api-access-5fwf9:{mountpoint:/var/lib/kubelet/pods/24c63ae0-ce91-4f48-9690-45033ad525cb/volumes/kubernetes.io~projected/kube-api-access-5fwf9 major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2df37093-9c54-4d41-ac8b-b41d8f83579d/volumes/kubernetes.io~projected/kube-api-access-6t4t9:{mountpoint:/var/lib/kubelet/pods/2df37093-9c54-4d41-ac8b-b41d8f83579d/volumes/kubernetes.io~projected/kube-api-access-6t4t9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/333ef57f-4887-4c8f-81c8-962265ad3be0/volumes/kubernetes.io~projected/kube-api-access-j57sb:{mountpoint:/var/lib/kubelet/pods/333ef57f-4887-4c8f-81c8-962265ad3be0/volumes/kubernetes.io~projected/kube-api-access-j57sb major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/333ef57f-4887-4c8f-81c8-962265ad3be0/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/333ef57f-4887-4c8f-81c8-962265ad3be0/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b7ef6fe-524b-4542-b281-27f7fce2deb8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3b7ef6fe-524b-4542-b281-27f7fce2deb8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3da65b23-4da0-4d6e-8475-0a5bc3c19768/volumes/kubernetes.io~projected/kube-api-access-2h4wn:{mountpoint:/var/lib/kubelet/pods/3da65b23-4da0-4d6e-8475-0a5bc3c19768/volumes/kubernetes.io~projected/kube-api-access-2h4wn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44590efe-c957-4995-8139-5b54567e51fd/volumes/kubernetes.io~projected/kube-api-access-86b25:{mountpoint:/var/lib/kubelet/pods/44590efe-c957-4995-8139-5b54567e51fd/volumes/kubernetes.io~projected/kube-api-access-86b25 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44590efe-c957-4995-8139-5b54567e51fd/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/44590efe-c957-4995-8139-5b54567e51fd/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4faac008-58e3-41b1-9f56-8a8760071532/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/4faac008-58e3-41b1-9f56-8a8760071532/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60d3aaa0-ccf0-42ae-92d5-44512e33aed4/volumes/kubernetes.io~projected/kube-api-access-pjcm6:{mountpoint:/var/lib/kubelet/pods/60d3aaa0-ccf0-42ae-92d5-44512e33aed4/volumes/kubernetes.io~projected/kube-api-access-pjcm6 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b75f9f-c202-4a6c-b370-c5ce3d1de74c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/66b75f9f-c202-4a6c-b370-c5ce3d1de74c/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/volumes/kubernetes.io~projected/kube-api-access-zp4fl:{mountpoint:/var/lib/kubelet/pods/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/volumes/kubernetes.io~projected/kube-api-access-zp4fl major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/757e39d9-7832-46de-8c68-930c0b80943e/volumes/kubernetes.io~projected/kube-api-access-wqtlv:{mountpoint:/var/lib/kubelet/pods/757e39d9-7832-46de-8c68-930c0b80943e/volumes/kubernetes.io~projected/kube-api-access-wqtlv major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/757e39d9-7832-46de-8c68-930c0b80943e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/757e39d9-7832-46de-8c68-930c0b80943e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75f38eb4-b4e8-4673-a540-24c8ebe84ee8/volumes/kubernetes.io~projected/kube-api-access-9bfhg:{mountpoint:/var/lib/kubelet/pods/75f38eb4-b4e8-4673-a540-24c8ebe84ee8/volumes/kubernetes.io~projected/kube-api-access-9bfhg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7978e0e4-9979-42cf-a15b-767a03352919/volumes/kubernetes.io~projected/kube-api-access-nnl8s:{mountpoint:/var/lib/kubelet/pods/7978e0e4-9979-42cf-a15b-767a03352919/volumes/kubernetes.io~projected/kube-api-access-nnl8s major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82de9774-f285-46a2-a1d0-cf66b6f780b2/volumes/kubernetes.io~projected/kube-api-access-blhsm:{mountpoint:/var/lib/kubelet/pods/82de9774-f285-46a2-a1d0-cf66b6f780b2/volumes/kubernetes.io~projected/kube-api-access-blhsm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8370c5cf-f442-4e7d-b8ab-ea6ede18552b/volumes/kubernetes.io~projected/kube-api-access-qqfsd:{mountpoint:/var/lib/kubelet/pods/8370c5cf-f442-4e7d-b8ab-ea6ede18552b/volumes/kubernetes.io~projected/kube-api-access-qqfsd major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8370c5cf-f442-4e7d-b8ab-ea6ede18552b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8370c5cf-f442-4e7d-b8ab-ea6ede18552b/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~projected/kube-api-access-2z9kl:{mountpoint:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~projected/kube-api-access-2z9kl major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/default-certificate major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/metrics-certs major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/844916e1-ce51-4692-907d-f30d1b6a9eb8/volumes/kubernetes.io~projected/kube-api-access-w7xvc:{mountpoint:/var/lib/kubelet/pods/844916e1-ce51-4692-907d-f30d1b6a9eb8/volumes/kubernetes.io~projected/kube-api-access-w7xvc major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volumes/kubernetes.io~projected/kube-api-access-hd55j:{mountpoint:/var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volumes/kubernetes.io~projected/kube-api-access-hd55j major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ba183e8-f0a3-41fa-9247-20576448a07f/volumes/kubernetes.io~projected/kube-api-access-r5xqn:{mountpoint:/var/lib/kubelet/pods/8ba183e8-f0a3-41fa-9247-20576448a07f/volumes/kubernetes.io~projected/kube-api-access-r5xqn major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/volumes/kubernetes.io~projected/kube-api-access-f76vc:{mountpoint:/var/lib/kubelet/pods/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/volumes/kubernetes.io~projected/kube-api-access-f76vc major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/volumes/kubernetes.io~secret/metrics-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb73706-3f89-4a36-83fa-8e5b42d55175/volumes/kubernetes.io~projected/kube-api-access-55szz:{mountpoint:/var/lib/kubelet/pods/9cb73706-3f89-4a36-83fa-8e5b42d55175/volumes/kubernetes.io~projected/kube-api-access-55szz major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5c72c64-5332-4820-870d-8a38037b2604/volumes/kubernetes.io~projected/kube-api-access-6qz2s:{mountpoint:/var/lib/kubelet/pods/a5c72c64-5332-4820-870d-8a38037b2604/volumes/kubernetes.io~projected/kube-api-access-6qz2s major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/kube-api-access-qxn75:{mountpoint:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/kube-api-access-qxn75 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/registry-tls major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07a2feb-3648-4f19-9385-baeb5eeeaeeb/volumes/kubernetes.io~projected/kube-api-access-vxjlq:{mountpoint:/var/lib/kubelet/pods/f07a2feb-3648-4f19-9385-baeb5eeeaeeb/volumes/kubernetes.io~projected/kube-api-access-vxjlq major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07a2feb-3648-4f19-9385-baeb5eeeaeeb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f07a2feb-3648-4f19-9385-baeb5eeeaeeb/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/01261e00581b2903cf767f52a017c4913e86451b1426ca01834a0f2a265edb36/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9c9532961076609944642bdbbcd2fbc1a733ce051817f7d4e5dd7bbced9ced75/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/cc9bb8f168096fc768c113078019312609a80a0760c64dede59dfac829e09cd4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2b2376e8081c4d97ce58d45e81e9cb0b2292652b30a9ef81467eae4e15fc8822/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/85a8848d5a636c63bc6b5954c1223d6d32df7a1673eeb4225fcea533fe22aa7f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8d8a05b6c7296531e0006d0e208c263a17de356f215fdb36d6ac426fb4678c65/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/a7de437142962f3247d2d3650f23c0fa0400aa937ffb6445b2be4cc81ecc8f68/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e3c07aa8c89e0bf2a8dcba73089740477bb86c486131cda73926d0d68fa64945/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/019012c0062622f8b192f7e7fc8e8363f8131315f866fd1072bbf41551efccdf/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/b2475716b8a8d09daec7f59df3cfbbf4e1a8e616d645380012638db3b41314e7/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/dc9a52461e5c1f1fa80d745a5411c132a4e757cc2d3a8d0a3d5d49ff6f39ead5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/85ccbb05a4525bdd0e070746f9b073a8fcdc5edb5977070c2caf101e83e4f202/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8e7c89baa863177c9d6b23f48f5024b3f00cf954db1b33640116b3e1a2ed8910/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e6bfb71d8ef67074aff90c0569cd236a608f1c2abd400a14567def615f6c89f1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3ce27a8b0c4bca61e52bfff2acbc48c3484072091aa2a7ac120826a18b20144b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/84bd792a45a9a838d48431b5d01b29b91fcacf18f2246b64ff947cd8d2fe1b7b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6aa06bbdb9a8f1f9cb99ab862e0c6a006ea8d94f2218aad280c13fd1c1a1ded1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6be80b8e2f0ec68d813e75a7387df743d891d6c799791f4725527d84ba692650/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4f2cba571613f5799c50a5ea457ec9145624d1dc8d86f2487b44e4d5c54fd928/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7184a1e090586fca27491fcf3bed02eafd05b53f733bb9a0be6ac06e7dbbf97f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/88a528768646ba5e97c295385628e48b8028cb0b5849175c50a7ab2ba57f4ab0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a9a2f6cd0b634b13bb1786bf93715661608353b3ae55fa939808eb89c3e7cd67/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8b9820086f4f48962313406781e8e69a73cd9f6108af13fe55b28b839aa7412b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/2ef3e78357091ed9975a5951a2a1f147a127f405e18524021211f5cefcd792a9/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/49053ad7828dc664552d661857830c543556429386af0c48843cadab3b523256/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/bc26796d988812a1c1e05a667736c12f9699283d9bd64173d7e9784b30b62c2e/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1afc6966a9554e837ddca6daab4767b083fb79c22ba41197e472c80c0472673e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/698dc379360be24b2ebd884974faa8d9c5c892452b88208a349e1625a5ad7dc5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/42373cdc87a2a68ef2ba9244f2c90e67a44bd7ff74706bd2938f11efd34341d4/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5bb16ca68ef152692165aea220abd171484971ed7839b8a9fb6a46896e42c2a8/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/b2fa44d23c2ed827cceb5d290f342f1506c3b815e4d1a3937b9c976f99d3ac80/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/870377dfcccd3f294b2a7c84c935696db71c5c7ba0dbb4a45e5b982604049890/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6fdd52ea1f5745b6ef92dedc87c89c9118ab8dcbe524fc9b2550cdaacbc1eb38/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/0baa0cc1e5cef80a837ed3f8670ea254a2858425969e31a8c87719a1120d59eb/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/f79d428532b8656d16bdcc8cafb09718ecb45b7a380bf1fd878ac830224930ee/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/3f04c19ab1ad402dda7050c0e733a64c787debfb1c50cc160fa6b4d17d72b13b/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/deaf131aedd458726f981de3a10a7998367a91229464615a82ff71ba74144eaf/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/e40c920568eea51da19d383fa85c362771a69b162fca6d8aee8aa7ae5a062ec4/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/07e21a96d8a75d2934441f9ca339669b4705ea37601931caa7485073336406b5/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/a72d0d7bfae7819824b1ffbb45f602891eac3a113b148eb9a6ce1367c90bf3af/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/68a5c06e4854dd6b5f4fcb042088556fff68c51c4a0d5abd818bd542f9b9d58c/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/0fbdce35c5189def62242b0f5a03ee8005d1c72e86e72fb804769c20f6f72b14/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/e3657bdb722e299c1f107e9095c2e0e622dbf7f9f000ce188e1d0253b8105849/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/7496b8861f7b30a7ac9dea1dca8ebf16735f2500d3dc8ff78442a3f5499651e3/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/d0c74a4271f8c51619aba04d7518dbae5d3398cf4486195060430861d099f3e6/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/b22e94a7f97df1cbeacb1dadd68957093fdbd74c2c0b554f58e62165b080341e/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/aa5bcb51ba607c736fcf01ca1d78cb5b8fa5289c9cda2237e5492769bc471f5f/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/e087de7992c8613bbfe10ff3bf0f74446ef5343a844227eb3805e3f77a5d9e3b/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/9bdfc0e4adde603d1adb0925d4f059b89fcff32f894eb48fecd317af2a651f20/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/93a56622199ea6ff84851088c5cb39a451d5fe19b50fb029dcab76cfab4b8a49/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/b4f5645262d57eb887c1b79cd26e3362dcb948817ffa7ad8d19656fb9a554392/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/4de0a87b1023a27da5dfbc9f093529b3147c9e79d864436c5dd687e192e94d81/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/a6807809d5d9c9757afec47489cb3c6b3a21c8755e56de66e93ad1d871e814a5/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/7d0f550652fcd1398a095945e94c80140d66677fa33d32034f924b162b1f4b4b/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/4e08ce646786f1f2f4d872e5407e162a1aa58675a3902d11048d7663093825f0/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/ca0c30a4dc4c06c52dd2e1a8614ba369cbcd84d7f7915ea3ec53a18e549cf894/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/c879c5df1d8552d6cdf37a8e28d9a27ea1da3f00720856776678fcfc557918d8/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/ad98f07daf27dfcebd60dd49d6db36300f856da28e2de730095526d6930df717/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/1d785fdc0d9e152d7868ee877f0c6da8db260c240e6a81e6e56468e892e3e3cb/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/fb46891382fbc3a992b183fe4dea45fe7857a6d350785b4a78f433caca0470ad/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/58e592f43392004b84ae56516ac13b320aedf361d8ca6253fc1067b90d054db4/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/11a6c05792a5ae35fa1687d70fe7dd83c462c9fcaea118946d407800a16f2a99/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/881a3bfd687d54217a379e7a9dcddad11af7ac27080e2f6ccc16b8d11bacde52/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/64904950c7ee1a65988ed4cfc9ea34cbf35baccc056a237b5a0646747314177a/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/cc885c1b84e0d93aa7ff0ca1406280b68d64eedbbfcf8653bfd4f66191741f21/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/1f2da3cb33cc8c4cf3244cd501e3b1e7a66ea433c95e2da60691f1b32c6ca330/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/e644a0278208bda76c4153c776532cbccc846b9e7c00d2bc5a2d8caa42b5d3d1/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/b4b2dfe8a122533090668505ceb3732fefb0ddcbdf8e3eb4f98ce763b907a5b4/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/51fa57db4363c32d935defc047f036654a2d329c6b9a518d6466e36913bfb7f5/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/1885df3fbf85650bff74f0a33b378a16fd18a5d7564e4a85838fe63b29f9b0dc/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/cb9a141fc499096ff77ff9716639f34cc3c521547ff0ec1cf3ac2cc84567fd75/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/21ebfb519df4e9aef174fd517d73ebef8b8f06f0180c1b4e0a2c9b6d72758d18/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c25e3925100bca01296fb3c002703a5c4a4a61bb465255e40cc444627c3da3d9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fd9489807f3167bf6ee0bec7b6442b092dd5fd2ead13b8c967517c493c5a4df0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/313772d4b9ffb79a1dcabd3cbc291bea7ec4196a441ae4da43d1970cd5722615/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/35fae18f57bc9840740b324a08bb734858631c0c818b9e31998e8fc9f094eb5e/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/31dccc434c61dfcd91767ceaa0457f232fcde7fe4f59c5ff7c0ce2e65ee61d8c/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8b8892254019849838ff3fbbb87f3303ea7e5ec23f82a06c09169b3a08ec1612/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8c5af6caf2f54e41bf72feb82d3828739d2bb6e4d709aa9f0fc7f7c9d56b4c9b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/82e7ac641a474b999f4146902f69e42e9bac7d808d7fa76a5ec70248a0802331/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4635294c96e9c8af4e58cb3c0ef0e560ad50fd4491fc0ea8a79b302bdf6fc1fa/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.529658 5815 manager.go:217] Machine: {Timestamp:2026-11-03 20:01:30.529183285 +0000 UTC m=+0.059528948 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3140262 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bc8b3541204e6f69f5c5c1a9f15a SystemUUID:ec24bc8b-3541-204e-6f69-f5c5c1a9f15a BootID:ec7131f1-6981-4c53-b926-086962e49496 Filesystems:[{Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b51ff3abef71de065c20d4f5d882768e4a5d5df4d8c8e8593daf70638d83793/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff22c92e9175e7597a63f5cf47a94301b8bdd704d2a7bb0ab547b7545dc6ad3c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57171b2cf7d2162d496d182d0db5d3cead5c3e621083a5e6af12b41310e8c438/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3da65b23-4da0-4d6e-8475-0a5bc3c19768/volumes/kubernetes.io~projected/kube-api-access-2h4wn DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed35a41a4635db4b29bb696539cc66db4f5aa2947d51d44ba736c2dc44971e0d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/333ef57f-4887-4c8f-81c8-962265ad3be0/volumes/kubernetes.io~projected/kube-api-access-j57sb DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44590efe-c957-4995-8139-5b54567e51fd/volumes/kubernetes.io~projected/kube-api-access-86b25 DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8370c5cf-f442-4e7d-b8ab-ea6ede18552b/volumes/kubernetes.io~projected/kube-api-access-qqfsd DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/60d3aaa0-ccf0-42ae-92d5-44512e33aed4/volumes/kubernetes.io~projected/kube-api-access-pjcm6 DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee4c1abc2ddfc4fd60293b8ae5bb6e65a57f8579d1f17362454d069cff72b0cf/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4debc594e5d3ad1758bd3878559a49965fac9e0fddb92e0015e5a9057d459ec9/userdata/shm DeviceMajor:0 DeviceMinor:360 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/333ef57f-4887-4c8f-81c8-962265ad3be0/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0248b06800d71f8a9e56363a54f8f4aa95e9bef384cfc88418415951809aaca/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2df37093-9c54-4d41-ac8b-b41d8f83579d/volumes/kubernetes.io~projected/kube-api-access-6t4t9 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~projected/kube-api-access-2z9kl DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:426 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/75f38eb4-b4e8-4673-a540-24c8ebe84ee8/volumes/kubernetes.io~projected/kube-api-access-9bfhg DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70e780095cf91194ac1198126da2074c0f2d4e970ed2893e3700724c3bac11d8/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a8f7716fd42f65b7b96086f5e3fb66a57eae68b63a0ec6b13cf38b1ef68e04a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e/volumes/kubernetes.io~projected/kube-api-access-zp4fl DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e51ce010f9806859942f050bc7d4bd843a456c8d5871460710d6489da3c395b5/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17053989-3da4-4c70-92c3-949cd0b733ee/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02c8db60-a588-40e9-a3b7-ba1ac37a5d47/volumes/kubernetes.io~projected/kube-api-access-7kckq DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b8e508618e3b68549bbc77f352f5a2cdb0f5c6930e9e17c3ce473677319c1b9/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5c72c64-5332-4820-870d-8a38037b2604/volumes/kubernetes.io~projected/kube-api-access-6qz2s DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ed58f76-099a-498d-844d-0069904b507b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81f33476d9ea83fc71ae2dde2ad1362efb651e4b12082c9d774c2237e73a3263/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05df1c3085fec22f323e2e6f44290ef91f018ccc18ac7ab784958d106c8ffa22/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3a2852f38023431f9b2e0cae7b7497003102e384daa3cd897510fbc2c880102/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f07a2feb-3648-4f19-9385-baeb5eeeaeeb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/757e39d9-7832-46de-8c68-930c0b80943e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddce53de112207fec64848aecb6977d4822c74752c622b898a3313c8fb0f540c/userdata/shm DeviceMajor:0 DeviceMinor:408 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/44590efe-c957-4995-8139-5b54567e51fd/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:420 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94e80fb9acee00281b5210ae13f31ce58caa2856aec9baf13a0491e653cc0f07/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8370c5cf-f442-4e7d-b8ab-ea6ede18552b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/033c2d08-c440-495d-9957-0b7d41914e1a/volumes/kubernetes.io~projected/kube-api-access-r7nbn DeviceMajor:0 DeviceMinor:303 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ba183e8-f0a3-41fa-9247-20576448a07f/volumes/kubernetes.io~projected/kube-api-access-r5xqn DeviceMajor:0 DeviceMinor:323 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:429 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/844916e1-ce51-4692-907d-f30d1b6a9eb8/volumes/kubernetes.io~projected/kube-api-access-w7xvc DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/757e39d9-7832-46de-8c68-930c0b80943e/volumes/kubernetes.io~projected/kube-api-access-wqtlv DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c97290fd72d2d004e46a814a8b93dce0beb38c60082d4d57d28c8ded03b9f33e/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f833588834de38052d27149bbababd688b397edbc07c05420742515021f8c8d/userdata/shm DeviceMajor:0 DeviceMinor:427 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66b75f9f-c202-4a6c-b370-c5ce3d1de74c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1679fbb3-a322-4c15-9837-df1040cc3f01/volumes/kubernetes.io~projected/kube-api-access-hgbwh DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e485ef4c50a26cc2c599da3fbbed332da8f4248209c9152ea31e5dd26ba967c3/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/volumes/kubernetes.io~projected/kube-api-access-f76vc DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b8b7acb63f2a8b4ce3c91e6e98cd59cd3b9d8f2e298dca93526cb1c4cad50e3/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05ab659c-a2dc-4764-a537-4b86fba538d2/volumes/kubernetes.io~projected/kube-api-access-6mvpb DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4faac008-58e3-41b1-9f56-8a8760071532/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:359 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/836e527c80221b6c9aca5d2eb2dc7a4af4ae0017581d749884dd77a8af7af2e3/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef4e004fc09c99f44aa19badd5f7e93779e7ae8ecf6845249ae35683c24e804d/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/148af517-1bbd-404e-97da-cadd491b1ded/volumes/kubernetes.io~projected/kube-api-access-6m7rm DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f8a024cf132f63bfda06b5e3ef68e48f3bf5aeabdb9b3d105ae44e8c54537d1/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7978e0e4-9979-42cf-a15b-767a03352919/volumes/kubernetes.io~projected/kube-api-access-nnl8s DeviceMajor:0 DeviceMinor:404 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/82de9774-f285-46a2-a1d0-cf66b6f780b2/volumes/kubernetes.io~projected/kube-api-access-blhsm DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f07a2feb-3648-4f19-9385-baeb5eeeaeeb/volumes/kubernetes.io~projected/kube-api-access-vxjlq DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24c63ae0-ce91-4f48-9690-45033ad525cb/volumes/kubernetes.io~projected/kube-api-access-5fwf9 DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/837adf32-c68a-46ef-bca6-5b5364f74576/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:425 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/808e8f88af16159f7430514cdd550ed1ac67f8ed1b9deb3f5440947f8c3656b4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f2fadfa5580a834de049f331cb52f046e4c0113af41da9df254671a010bbc1ac/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02c8db60-a588-40e9-a3b7-ba1ac37a5d47/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3b7ef6fe-524b-4542-b281-27f7fce2deb8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volumes/kubernetes.io~projected/kube-api-access-hd55j DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/849c8451-1aac-44aa-a685-28b03801682e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~projected/kube-api-access-qxn75 DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb73706-3f89-4a36-83fa-8e5b42d55175/volumes/kubernetes.io~projected/kube-api-access-55szz DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3edb09c604bd7b56aa109dde93c7b54eaa04361d45ac9a599918194cc92b03f/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c40ebe22870af14cce1967ec757c4dc615ffa82c12d91eba121fcdf1e15ec5f7/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17053989-3da4-4c70-92c3-949cd0b733ee/volumes/kubernetes.io~projected/kube-api-access-649bz DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1551a4b4-7bd1-47ec-ae80-e79f4262761e/volumes/kubernetes.io~projected/kube-api-access-9r6fx DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/deb2412ec78ed76c89400bf62fc2bb7e23b1df58bec9a4bf6453eee6830a57cb/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0ed58f76-099a-498d-844d-0069904b507b/volumes/kubernetes.io~projected/kube-api-access-qbz8z DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e26ba4e47a789134f357e05369b6e501fe0cec470a68f998cc90f7d0a46cfbb3/userdata/shm DeviceMajor:0 DeviceMinor:432 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1b8b7acb63f2a8b MacAddress:ea:be:cf:7f:39:43 Speed:10000 Mtu:8901} {Name:3b8e508618e3b68 MacAddress:42:50:b4:42:a9:58 Speed:10000 Mtu:8901} {Name:4debc594e5d3ad1 MacAddress:3a:60:c8:15:e9:7f Speed:10000 Mtu:8901} {Name:57171b2cf7d2162 MacAddress:c6:91:93:ed:fa:e5 Speed:10000 Mtu:8901} {Name:6bb36c71cef7423 MacAddress:2a:52:14:f0:9c:14 Speed:10000 Mtu:8901} {Name:70e780095cf9119 MacAddress:a2:a2:a8:e0:f0:ee Speed:10000 Mtu:8901} {Name:81f33476d9ea83f MacAddress:1e:5f:ad:50:b2:90 Speed:10000 Mtu:8901} {Name:836e527c80221b6 MacAddress:2e:4a:97:b7:47:51 Speed:10000 Mtu:8901} {Name:8f8a024cf132f63 MacAddress:46:db:e6:0c:ba:cd Speed:10000 Mtu:8901} {Name:9f833588834de38 MacAddress:c6:a9:09:df:70:ee Speed:10000 Mtu:8901} {Name:b70e4806378420d MacAddress:ee:69:86:3c:79:d6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fb:22:f9:09 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:74:b3:95:99:7e Speed:0 Mtu:8901} {Name:c0248b06800d71f MacAddress:d2:9d:29:2d:77:d8 Speed:10000 Mtu:8901} {Name:c40ebe22870af14 MacAddress:4e:87:e9:5f:51:cd Speed:10000 Mtu:8901} {Name:c97290fd72d2d00 MacAddress:e2:c4:7a:8a:af:e6 Speed:10000 Mtu:8901} {Name:ddce53de112207f MacAddress:9e:3e:55:5a:44:33 Speed:10000 Mtu:8901} {Name:deb2412ec78ed76 MacAddress:56:b7:8f:2e:36:1e Speed:10000 Mtu:8901} {Name:e26ba4e47a78913 MacAddress:72:10:a2:98:61:57 Speed:10000 Mtu:8901} {Name:e3a2852f3802343 MacAddress:92:8e:c7:f9:c0:e7 Speed:10000 Mtu:8901} {Name:e485ef4c50a26cc MacAddress:42:e2:16:51:c2:d0 Speed:10000 Mtu:8901} {Name:e51ce010f980685 MacAddress:fa:a8:d6:1d:0d:be Speed:10000 Mtu:8901} {Name:ee4c1abc2ddfc4f MacAddress:aa:5a:65:3f:65:4f Speed:10000 Mtu:8901} {Name:ef4e004fc09c99f MacAddress:52:4d:1b:b3:d0:13 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fb:22:f9:09 Speed:0 Mtu:9001} {Name:f2fadfa5580a834 MacAddress:22:11:82:c1:c2:80 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2e:b3:24:b9:89:88 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0a:cc:b2:7e:69:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530074 5815 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530115 5815 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530381 5815 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530407 5815 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-81.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530544 5815 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530552 5815 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530567 5815 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530579 5815 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530653 5815 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:01:30.530875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530799 5815 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:01:30.531461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530891 5815 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:01:30.531461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530924 5815 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:01:30.531461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530937 5815 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:01:30.531461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530947 5815 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:01:30.531461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530955 5815 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:01:30.531461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.530962 5815 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:01:30.531725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.531710 5815 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:01:30.531760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.531726 5815 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:01:30.532217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.532198 5815 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:01:30.535278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535265 5815 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:01:30.535278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535279 5815 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:01:30.535622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535371 5815 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:01:30.535622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535589 5815 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:01:30.535622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535617 5815 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:01:30.535729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535718 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:01:30.535763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535734 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:01:30.535763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535740 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:01:30.535763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535746 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:01:30.535763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535757 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535766 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535775 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535783 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535790 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535796 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535810 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535819 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:01:30.535867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535843 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:01:30.536099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.535873 5815 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:01:30.539436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.539421 5815 server.go:1263] "Started kubelet" Nov 03 20:01:30.539558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.539496 5815 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:01:30.539622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.539580 5815 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:01:30.540321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.539686 5815 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:01:30.540321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.540009 5815 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:01:30.539996 ip-10-0-143-81 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:01:30.547543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.542802 5815 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:01:30.547955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.547937 5815 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.547958 5815 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.548006 5815 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:19 +0000 UTC" deadline="2028-05-08 20:15:30.944743828 +0000 UTC" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.548022 5815 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13248h14m0.396724636s" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.548025 5815 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-81.us-east-2.compute.internal\" not found" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.548065 5815 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.548072 5815 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:01:30.548396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.548133 5815 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:01:30.549168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549148 5815 factory.go:55] Registering systemd factory Nov 03 20:01:30.549168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549169 5815 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:01:30.549704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549680 5815 factory.go:153] Registering CRI-O factory Nov 03 20:01:30.549704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549700 5815 factory.go:223] Registration of the crio container factory successfully Nov 03 20:01:30.549858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549757 5815 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:01:30.549858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549776 5815 factory.go:103] Registering Raw factory Nov 03 20:01:30.549858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.549794 5815 manager.go:1199] Started watching for new ooms in manager Nov 03 20:01:30.550167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550154 5815 manager.go:319] Starting recovery of all containers Nov 03 20:01:30.550781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550752 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5c72c64-5332-4820-870d-8a38037b2604" volumeName="kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550782 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550791 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550799 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c8db60-a588-40e9-a3b7-ba1ac37a5d47" volumeName="kubernetes.io/projected/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-kube-api-access-7kckq" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550806 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1551a4b4-7bd1-47ec-ae80-e79f4262761e" volumeName="kubernetes.io/projected/1551a4b4-7bd1-47ec-ae80-e79f4262761e-kube-api-access-9r6fx" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550812 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4faac008-58e3-41b1-9f56-8a8760071532" volumeName="kubernetes.io/configmap/4faac008-58e3-41b1-9f56-8a8760071532-nginx-conf" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550819 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" volumeName="kubernetes.io/secret/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-serving-cert" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550827 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="148af517-1bbd-404e-97da-cadd491b1ded" volumeName="kubernetes.io/projected/148af517-1bbd-404e-97da-cadd491b1ded-kube-api-access-6m7rm" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550835 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17053989-3da4-4c70-92c3-949cd0b733ee" volumeName="kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550841 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333ef57f-4887-4c8f-81c8-962265ad3be0" volumeName="kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-trusted-ca" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550849 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8370c5cf-f442-4e7d-b8ab-ea6ede18552b" volumeName="kubernetes.io/configmap/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-config" seLinuxMountContext="" Nov 03 20:01:30.550849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550856 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="844916e1-ce51-4692-907d-f30d1b6a9eb8" volumeName="kubernetes.io/projected/844916e1-ce51-4692-907d-f30d1b6a9eb8-kube-api-access-w7xvc" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550863 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="849c8451-1aac-44aa-a685-28b03801682e" volumeName="kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550870 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550890 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" volumeName="kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550897 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c63ae0-ce91-4f48-9690-45033ad525cb" volumeName="kubernetes.io/projected/24c63ae0-ce91-4f48-9690-45033ad525cb-kube-api-access-5fwf9" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550917 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8370c5cf-f442-4e7d-b8ab-ea6ede18552b" volumeName="kubernetes.io/secret/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-serving-cert" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550924 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" volumeName="kubernetes.io/projected/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-kube-api-access-vxjlq" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550930 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c8db60-a588-40e9-a3b7-ba1ac37a5d47" volumeName="kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550947 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f38eb4-b4e8-4673-a540-24c8ebe84ee8" volumeName="kubernetes.io/configmap/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-serviceca" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550954 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="837adf32-c68a-46ef-bca6-5b5364f74576" volumeName="kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550961 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5c72c64-5332-4820-870d-8a38037b2604" volumeName="kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550967 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44590efe-c957-4995-8139-5b54567e51fd" volumeName="kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550974 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82de9774-f285-46a2-a1d0-cf66b6f780b2" volumeName="kubernetes.io/projected/82de9774-f285-46a2-a1d0-cf66b6f780b2-kube-api-access-blhsm" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550982 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8370c5cf-f442-4e7d-b8ab-ea6ede18552b" volumeName="kubernetes.io/projected/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-kube-api-access-qqfsd" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550990 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5c72c64-5332-4820-870d-8a38037b2604" volumeName="kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-binary-copy" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.550997 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" volumeName="kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-snapshots" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551003 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3da65b23-4da0-4d6e-8475-0a5bc3c19768" volumeName="kubernetes.io/projected/3da65b23-4da0-4d6e-8475-0a5bc3c19768-kube-api-access-2h4wn" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551013 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44590efe-c957-4995-8139-5b54567e51fd" volumeName="kubernetes.io/projected/44590efe-c957-4995-8139-5b54567e51fd-kube-api-access-86b25" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551020 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7978e0e4-9979-42cf-a15b-767a03352919" volumeName="kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551027 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333ef57f-4887-4c8f-81c8-962265ad3be0" volumeName="kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-config" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551034 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333ef57f-4887-4c8f-81c8-962265ad3be0" volumeName="kubernetes.io/projected/333ef57f-4887-4c8f-81c8-962265ad3be0-kube-api-access-j57sb" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551049 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b75f9f-c202-4a6c-b370-c5ce3d1de74c" volumeName="kubernetes.io/secret/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-agent-certs" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551056 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="837adf32-c68a-46ef-bca6-5b5364f74576" volumeName="kubernetes.io/projected/837adf32-c68a-46ef-bca6-5b5364f74576-kube-api-access-2z9kl" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551064 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e197f7-de69-4fb1-b6d8-e1ffaa6d4903" volumeName="kubernetes.io/configmap/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-config-volume" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551071 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551078 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2df37093-9c54-4d41-ac8b-b41d8f83579d" volumeName="kubernetes.io/projected/2df37093-9c54-4d41-ac8b-b41d8f83579d-kube-api-access-6t4t9" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551085 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714f1a1a-a854-4a9c-a2f6-d07990fe9f0e" volumeName="kubernetes.io/configmap/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-config" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551092 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82de9774-f285-46a2-a1d0-cf66b6f780b2" volumeName="kubernetes.io/empty-dir/82de9774-f285-46a2-a1d0-cf66b6f780b2-tmp-dir" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551099 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ba183e8-f0a3-41fa-9247-20576448a07f" volumeName="kubernetes.io/projected/8ba183e8-f0a3-41fa-9247-20576448a07f-kube-api-access-r5xqn" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551107 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e197f7-de69-4fb1-b6d8-e1ffaa6d4903" volumeName="kubernetes.io/empty-dir/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-tmp-dir" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551114 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e197f7-de69-4fb1-b6d8-e1ffaa6d4903" volumeName="kubernetes.io/projected/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-kube-api-access-f76vc" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551122 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02c8db60-a588-40e9-a3b7-ba1ac37a5d47" volumeName="kubernetes.io/configmap/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-config" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551131 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ab659c-a2dc-4764-a537-4b86fba538d2" volumeName="kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-tuned" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551138 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ed58f76-099a-498d-844d-0069904b507b" volumeName="kubernetes.io/secret/0ed58f76-099a-498d-844d-0069904b507b-cert" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551144 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1679fbb3-a322-4c15-9837-df1040cc3f01" volumeName="kubernetes.io/projected/1679fbb3-a322-4c15-9837-df1040cc3f01-kube-api-access-hgbwh" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551151 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551159 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1679fbb3-a322-4c15-9837-df1040cc3f01" volumeName="kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-daemon-config" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551166 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714f1a1a-a854-4a9c-a2f6-d07990fe9f0e" volumeName="kubernetes.io/secret/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-serving-cert" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551174 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ab659c-a2dc-4764-a537-4b86fba538d2" volumeName="kubernetes.io/projected/05ab659c-a2dc-4764-a537-4b86fba538d2-kube-api-access-6mvpb" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551182 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ed58f76-099a-498d-844d-0069904b507b" volumeName="kubernetes.io/projected/0ed58f76-099a-498d-844d-0069904b507b-kube-api-access-qbz8z" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551189 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17053989-3da4-4c70-92c3-949cd0b733ee" volumeName="kubernetes.io/projected/17053989-3da4-4c70-92c3-949cd0b733ee-kube-api-access-649bz" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551196 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714f1a1a-a854-4a9c-a2f6-d07990fe9f0e" volumeName="kubernetes.io/projected/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-kube-api-access-zp4fl" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551203 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="757e39d9-7832-46de-8c68-930c0b80943e" volumeName="kubernetes.io/projected/757e39d9-7832-46de-8c68-930c0b80943e-kube-api-access-wqtlv" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551210 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e197f7-de69-4fb1-b6d8-e1ffaa6d4903" volumeName="kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551218 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551230 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" volumeName="kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-tmp" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551237 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="033c2d08-c440-495d-9957-0b7d41914e1a" volumeName="kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551245 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1679fbb3-a322-4c15-9837-df1040cc3f01" volumeName="kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-cni-binary-copy" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551255 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333ef57f-4887-4c8f-81c8-962265ad3be0" volumeName="kubernetes.io/secret/333ef57f-4887-4c8f-81c8-962265ad3be0-serving-cert" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551266 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4faac008-58e3-41b1-9f56-8a8760071532" volumeName="kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551278 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="757e39d9-7832-46de-8c68-930c0b80943e" volumeName="kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551291 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5c72c64-5332-4820-870d-8a38037b2604" volumeName="kubernetes.io/projected/a5c72c64-5332-4820-870d-8a38037b2604-kube-api-access-6qz2s" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551310 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60d3aaa0-ccf0-42ae-92d5-44512e33aed4" volumeName="kubernetes.io/projected/60d3aaa0-ccf0-42ae-92d5-44512e33aed4-kube-api-access-pjcm6" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551322 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="837adf32-c68a-46ef-bca6-5b5364f74576" volumeName="kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551333 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551343 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="837adf32-c68a-46ef-bca6-5b5364f74576" volumeName="kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-default-certificate" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551356 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="033c2d08-c440-495d-9957-0b7d41914e1a" volumeName="kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551374 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17053989-3da4-4c70-92c3-949cd0b733ee" volumeName="kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551384 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3da65b23-4da0-4d6e-8475-0a5bc3c19768" volumeName="kubernetes.io/configmap/3da65b23-4da0-4d6e-8475-0a5bc3c19768-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551396 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75f38eb4-b4e8-4673-a540-24c8ebe84ee8" volumeName="kubernetes.io/projected/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-kube-api-access-9bfhg" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551406 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="837adf32-c68a-46ef-bca6-5b5364f74576" volumeName="kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-stats-auth" seLinuxMountContext="" Nov 03 20:01:30.551377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551420 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="849c8451-1aac-44aa-a685-28b03801682e" volumeName="kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-env-overrides" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551431 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="849c8451-1aac-44aa-a685-28b03801682e" volumeName="kubernetes.io/projected/849c8451-1aac-44aa-a685-28b03801682e-kube-api-access-hd55j" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551448 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="033c2d08-c440-495d-9957-0b7d41914e1a" volumeName="kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551459 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17053989-3da4-4c70-92c3-949cd0b733ee" volumeName="kubernetes.io/empty-dir/17053989-3da4-4c70-92c3-949cd0b733ee-data-volume" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551471 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44590efe-c957-4995-8139-5b54567e51fd" volumeName="kubernetes.io/configmap/44590efe-c957-4995-8139-5b54567e51fd-telemetry-config" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551481 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="849c8451-1aac-44aa-a685-28b03801682e" volumeName="kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-config" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551492 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="849c8451-1aac-44aa-a685-28b03801682e" volumeName="kubernetes.io/secret/849c8451-1aac-44aa-a685-28b03801682e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551521 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b843f245-431e-40af-a566-a4fa62de8c39" volumeName="kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551533 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" volumeName="kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-service-ca-bundle" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551544 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="033c2d08-c440-495d-9957-0b7d41914e1a" volumeName="kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551558 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ab659c-a2dc-4764-a537-4b86fba538d2" volumeName="kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-tmp" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551574 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b7ef6fe-524b-4542-b281-27f7fce2deb8" volumeName="kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551585 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b75f9f-c202-4a6c-b370-c5ce3d1de74c" volumeName="kubernetes.io/configmap/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-konnectivity-ca" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551598 5815 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb73706-3f89-4a36-83fa-8e5b42d55175" volumeName="kubernetes.io/projected/9cb73706-3f89-4a36-83fa-8e5b42d55175-kube-api-access-55szz" seLinuxMountContext="" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551609 5815 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:01:30.553280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.551616 5815 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:01:30.554277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.554259 5815 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:30.555348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.555326 5815 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:30.557403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.557381 5815 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:30.558223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.558206 5815 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:01:30.568076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.568053 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:01:30.569646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.569631 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:01:30.575085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.575069 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:01:30.575452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.575395 5815 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:01:30.576419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.576398 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:01:30.577212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.577192 5815 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:01:30.577212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.577214 5815 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:01:30.577353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.577232 5815 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:01:30.577353 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.577273 5815 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:01:30.581732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.581710 5815 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:30.605654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.605637 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:01:30.605758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.605664 5815 generic.go:356] "Generic (PLEG): container finished" podID="333ef57f-4887-4c8f-81c8-962265ad3be0" containerID="c0952f28baf936512f566d16da0236fd83f8112e3a0beebedfa0c585a9628b48" exitCode=255 Nov 03 20:01:30.609148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.609123 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="2ca7cc5f999570d4ec7eb09ac690deda64fac08e065706ca16ab6124566dc13d" exitCode=0 Nov 03 20:01:30.609226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.609150 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="07b2202d1158d8e7ed0c74cd50129f0634c53d48dddb806c62daa355d80f546c" exitCode=0 Nov 03 20:01:30.609226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.609161 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="9818d7b0901142a305d2100818fdbc2c7f2490c90ebe2132e9d5bae6a06209a1" exitCode=0 Nov 03 20:01:30.609226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.609171 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="d628f356bcb9a14b06df6b67cee2c88bff87b41a66b340846be40ca40ff4928d" exitCode=0 Nov 03 20:01:30.609226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.609183 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="a121d8dce8715d13dfc017e2177b56e528163cd3aab6b84d026c4ab87eab5fc8" exitCode=0 Nov 03 20:01:30.609226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.609192 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c72c64-5332-4820-870d-8a38037b2604" containerID="a1536072e67912747fb2dcdb651087c06da274da344d14e7bfa8a9101a855a1f" exitCode=0 Nov 03 20:01:30.613852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.613827 5815 generic.go:356] "Generic (PLEG): container finished" podID="c70b0209b579d0e0dc902659c53a82cd" containerID="af4ca698098c9dc4eee9927264849323d3c4b33ff0a03a5f900ffed288cf74bf" exitCode=0 Nov 03 20:01:30.645365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.645316 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:01:30.645725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.645706 5815 generic.go:356] "Generic (PLEG): container finished" podID="849c8451-1aac-44aa-a685-28b03801682e" containerID="5ff47328f0b3a6fee0960dedc7cd8fb4a3411888c8860cc70114e6b41eaf7045" exitCode=1 Nov 03 20:01:30.667829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.667807 5815 manager.go:324] Recovery completed Nov 03 20:01:30.677374 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.677350 5815 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 20:01:30.690681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690664 5815 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:01:30.690681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690677 5815 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:01:30.690802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690692 5815 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:01:30.690802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690793 5815 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:01:30.690860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690803 5815 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:01:30.690860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690819 5815 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:01:30.690860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690825 5815 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:01:30.690860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690829 5815 policy_none.go:50] "Start" Nov 03 20:01:30.690860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690837 5815 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:01:30.690860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690845 5815 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:01:30.691012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690969 5815 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:01:30.691012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690976 5815 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:01:30.691012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.690981 5815 policy_none.go:44] "Start" Nov 03 20:01:30.695685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.695655 5815 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:01:30.698284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.698272 5815 manager.go:341] "Starting Device Plugin manager" Nov 03 20:01:30.698852 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.698295 5815 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:01:30.698852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.698854 5815 server.go:87] "Starting device plugin registration server" Nov 03 20:01:30.699093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699083 5815 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:01:30.699141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699094 5815 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:01:30.699230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699205 5815 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:01:30.699367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699330 5815 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:01:30.699441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699378 5815 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:01:30.699441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699386 5815 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:01:30.699561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.699436 5815 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:01:30.699365409Z","UUID":"7633795f-40dd-4cbc-b1e4-f2b387b7b33b","Handler":null,"Name":"","Endpoint":""} Nov 03 20:01:30.701754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.701732 5815 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:01:30.701906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.701773 5815 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:01:30.702866 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.702848 5815 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:01:30.799743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.799724 5815 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:01:30.801640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.801624 5815 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:01:30.801701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.801655 5815 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:01:30.801701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.801671 5815 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-81.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:01:30.801759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.801730 5815 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.816199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.816184 5815 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.816315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.816236 5815 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.877811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.877774 5815 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal"] Nov 03 20:01:30.878399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878368 5815 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.878443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878376 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"c70b0209b579d0e0dc902659c53a82cd","Type":"ContainerStarted","Data":"93b6694059b8570f5ad5d114da9302e98725d172cfed095fe813024617badd99"} Nov 03 20:01:30.878443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878436 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"c70b0209b579d0e0dc902659c53a82cd","Type":"ContainerDied","Data":"af4ca698098c9dc4eee9927264849323d3c4b33ff0a03a5f900ffed288cf74bf"} Nov 03 20:01:30.878518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878449 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"c70b0209b579d0e0dc902659c53a82cd","Type":"ContainerStarted","Data":"ccefb5ff365b8079a0ab76ac5c8b250923b344a2b2f30083e43c0d9ece6499c1"} Nov 03 20:01:30.878518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878473 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"7974090a1f6621f0627618a748dcd304","Type":"ContainerStarted","Data":"1e9041bcf95320e2bfb06c672584685a9fff8ed4cad7e86ab2cf0a69ab7e8bda"} Nov 03 20:01:30.878518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878491 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" event={"ID":"7974090a1f6621f0627618a748dcd304","Type":"ContainerStarted","Data":"5e4687db94f742b67a2d6a450abfd28cab5ab9615ca0566df83b81bf80a82d9e"} Nov 03 20:01:30.878663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.878648 5815 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.889234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.889213 5815 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:30.889314 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.889254 5815 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.889859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.889847 5815 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:30.889927 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:30.889881 5815 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.960190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.960169 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.960277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.960201 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:30.960277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:30.960224 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7974090a1f6621f0627618a748dcd304-config\") pod \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"7974090a1f6621f0627618a748dcd304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.060795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.060762 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.060911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.060799 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.060911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.060821 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.060911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.060842 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7974090a1f6621f0627618a748dcd304-config\") pod \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"7974090a1f6621f0627618a748dcd304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.060911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.060889 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7974090a1f6621f0627618a748dcd304-config\") pod \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"7974090a1f6621f0627618a748dcd304\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.061045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.060913 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c70b0209b579d0e0dc902659c53a82cd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" (UID: \"c70b0209b579d0e0dc902659c53a82cd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.531681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.531652 5815 apiserver.go:52] "Watching apiserver" Nov 03 20:01:31.542939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.542917 5815 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:01:31.545497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.545445 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mxp5k","openshift-multus/multus-88mp6","openshift-insights/periodic-gathering-g9msr-f58ll","openshift-network-console/networking-console-plugin-86c478bb88-zwgdc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw","openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2","openshift-insights/insights-operator-7c4986d449-hft4t","kube-system/kas-connection-checker-685895c7c9-7qnz8","kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh","openshift-ovn-kubernetes/ovnkube-node-dt8zf","openshift-dns/dns-default-d6zdd","kube-system/konnectivity-agent-cg74r","openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx","openshift-multus/multus-additional-cni-plugins-kvv5k","openshift-multus/network-metrics-daemon-bpqrl","openshift-network-operator/iptables-alerter-q4r6s","openshift-dns/node-resolver-j4p9c","kube-system/kas-connection-checker-685895c7c9-6kg75","openshift-cluster-node-tuning-operator/tuned-htdgg","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph","openshift-ingress/router-default-585db4bb4c-xs84v","kube-system/global-pull-secret-syncer-6k5bk","openshift-network-diagnostics/network-check-target-kglzd","openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj","openshift-image-registry/image-registry-5d87675b69-6m79x","openshift-ingress-canary/ingress-canary-mt748","openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll","openshift-insights/insights-runtime-extractor-9p8gl","openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t","openshift-image-registry/node-ca-n27x8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-fbn7r","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2"] Nov 03 20:01:31.551108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.551085 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.551238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.551222 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.551359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.551337 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:01:31.552079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.552064 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xfnff\"" Nov 03 20:01:31.552320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.552302 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:01:31.554659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.554639 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.554782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.554766 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.555118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555095 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:01:31.555118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555108 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.555278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555170 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:01:31.555278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555178 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:01:31.555278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555181 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.555278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555236 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.555278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555263 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.555628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555590 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:01:31.555628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555590 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.555754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555633 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:01:31.555754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555638 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.555754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555683 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.555754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.555728 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:01:31.556055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556036 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.556055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556051 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.556587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556572 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.556587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556581 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.556696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556640 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.556767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556714 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:01:31.556828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.556818 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.561982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.561956 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.562069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562006 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:01:31.562347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562328 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.562439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562425 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.562620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562519 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.562862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562843 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:01:31.562933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562872 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:01:31.563033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.562613 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.563555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.563536 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.563631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.563561 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:01:31.563738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.563719 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\"" Nov 03 20:01:31.563993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.563969 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:01:31.564067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564050 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zh9fq\"" Nov 03 20:01:31.564110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564084 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.564152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.563537 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.564329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564292 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjlq\" (UniqueName: \"kubernetes.io/projected/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-kube-api-access-vxjlq\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.564481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564365 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.564481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564401 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.564481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564440 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:01:31.564626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564601 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:01:31.564626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564643 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rfsd7\"" Nov 03 20:01:31.564626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.564655 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:01:31.565851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.565833 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:01:31.565925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.565909 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:01:31.565925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.565917 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:01:31.566042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566017 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnml9\"" Nov 03 20:01:31.566112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566064 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8fc4f\"" Nov 03 20:01:31.566180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566138 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f6jr5\"" Nov 03 20:01:31.566180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566147 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-458rx\"" Nov 03 20:01:31.566180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566149 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:01:31.566180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566173 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-serving-cert\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566193 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-grxf8\"" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566212 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55szz\" (UniqueName: \"kubernetes.io/projected/9cb73706-3f89-4a36-83fa-8e5b42d55175-kube-api-access-55szz\") pod \"network-check-source-6f78d59f99-6tmll\" (UID: \"9cb73706-3f89-4a36-83fa-8e5b42d55175\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566243 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44590efe-c957-4995-8139-5b54567e51fd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566279 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566301 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566311 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7rm\" (UniqueName: \"kubernetes.io/projected/148af517-1bbd-404e-97da-cadd491b1ded-kube-api-access-6m7rm\") pod \"kas-connection-checker-685895c7c9-7qnz8\" (UID: \"148af517-1bbd-404e-97da-cadd491b1ded\") " pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566353 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xvc\" (UniqueName: \"kubernetes.io/projected/844916e1-ce51-4692-907d-f30d1b6a9eb8-kube-api-access-w7xvc\") pod \"kas-connection-checker-685895c7c9-fbn7r\" (UID: \"844916e1-ce51-4692-907d-f30d1b6a9eb8\") " pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566374 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566388 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:01:31.566412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566404 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-r5sx5\"" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566448 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbl6p\"" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566386 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6fx\" (UniqueName: \"kubernetes.io/projected/1551a4b4-7bd1-47ec-ae80-e79f4262761e-kube-api-access-9r6fx\") pod \"kas-connection-checker-685895c7c9-6kg75\" (UID: \"1551a4b4-7bd1-47ec-ae80-e79f4262761e\") " pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566493 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-tmp\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566562 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566595 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-service-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566599 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-tmp\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566622 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-config\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566570 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whl79\"" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566649 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfsd\" (UniqueName: \"kubernetes.io/projected/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-kube-api-access-qqfsd\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566678 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b25\" (UniqueName: \"kubernetes.io/projected/44590efe-c957-4995-8139-5b54567e51fd-kube-api-access-86b25\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566703 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-snapshots\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566725 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-serving-cert\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566802 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566828 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-config\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566910 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566951 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566965 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/44590efe-c957-4995-8139-5b54567e51fd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:31.566984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.566652 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfqpt\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567012 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567054 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567139 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5fnjm\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567164 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s55dl\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567226 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lxhjg\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567278 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-service-ca-bundle\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567322 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ms2v5\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567336 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-snapshots\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567350 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567404 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-serving-cert\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567468 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567487 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567618 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567624 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/44590efe-c957-4995-8139-5b54567e51fd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567825 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567852 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567878 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567930 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vncq9\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.567946 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:01:31.568034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568012 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pw8jf\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568116 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568127 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568117 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568242 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568274 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568275 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568275 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568322 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568359 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568375 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568388 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568408 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kmt87\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568620 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568959 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:01:31.569001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.568998 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vlwgf\"" Nov 03 20:01:31.569727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.569033 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.569727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.569084 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lrvk5\"" Nov 03 20:01:31.570038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.570023 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:01:31.577180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.577156 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-serving-cert\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:01:31.577315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.577298 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mszh\"" Nov 03 20:01:31.577416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.577342 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:01:31.577467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.577425 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\"" Nov 03 20:01:31.577498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.577481 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:01:31.577627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.577569 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n4r7h\"" Nov 03 20:01:31.581939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.581921 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7rm\" (UniqueName: \"kubernetes.io/projected/148af517-1bbd-404e-97da-cadd491b1ded-kube-api-access-6m7rm\") pod \"kas-connection-checker-685895c7c9-7qnz8\" (UID: \"148af517-1bbd-404e-97da-cadd491b1ded\") " pod="kube-system/kas-connection-checker-685895c7c9-7qnz8" Nov 03 20:01:31.583520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.583483 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfsd\" (UniqueName: \"kubernetes.io/projected/8370c5cf-f442-4e7d-b8ab-ea6ede18552b-kube-api-access-qqfsd\") pod \"service-ca-operator-544d749c8c-j5jf2\" (UID: \"8370c5cf-f442-4e7d-b8ab-ea6ede18552b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" Nov 03 20:01:31.583520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.583497 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xvc\" (UniqueName: \"kubernetes.io/projected/844916e1-ce51-4692-907d-f30d1b6a9eb8-kube-api-access-w7xvc\") pod \"kas-connection-checker-685895c7c9-fbn7r\" (UID: \"844916e1-ce51-4692-907d-f30d1b6a9eb8\") " pod="kube-system/kas-connection-checker-685895c7c9-fbn7r" Nov 03 20:01:31.583776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.583761 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b25\" (UniqueName: \"kubernetes.io/projected/44590efe-c957-4995-8139-5b54567e51fd-kube-api-access-86b25\") pod \"cluster-monitoring-operator-884576c57-txxdx\" (UID: \"44590efe-c957-4995-8139-5b54567e51fd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-txxdx" Nov 03 20:01:31.583849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.583826 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjlq\" (UniqueName: \"kubernetes.io/projected/f07a2feb-3648-4f19-9385-baeb5eeeaeeb-kube-api-access-vxjlq\") pod \"insights-operator-7c4986d449-hft4t\" (UID: \"f07a2feb-3648-4f19-9385-baeb5eeeaeeb\") " pod="openshift-insights/insights-operator-7c4986d449-hft4t" Nov 03 20:01:31.583886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.583825 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55szz\" (UniqueName: \"kubernetes.io/projected/9cb73706-3f89-4a36-83fa-8e5b42d55175-kube-api-access-55szz\") pod \"network-check-source-6f78d59f99-6tmll\" (UID: \"9cb73706-3f89-4a36-83fa-8e5b42d55175\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-6tmll" Nov 03 20:01:31.583960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.583946 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6fx\" (UniqueName: \"kubernetes.io/projected/1551a4b4-7bd1-47ec-ae80-e79f4262761e-kube-api-access-9r6fx\") pod \"kas-connection-checker-685895c7c9-6kg75\" (UID: \"1551a4b4-7bd1-47ec-ae80-e79f4262761e\") " pod="kube-system/kas-connection-checker-685895c7c9-6kg75" Nov 03 20:01:31.650030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.650009 5815 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:01:31.650968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.650945 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:01:31.651080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.651030 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:01:31.651080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.651038 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.651165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.651105 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.654934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.654919 5815 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.655042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.655028 5815 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.667653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667628 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-lib-modules\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.667738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667663 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kckq\" (UniqueName: \"kubernetes.io/projected/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-kube-api-access-7kckq\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:31.667738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667691 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-etc-kubernetes\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.667801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667735 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.667918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667899 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-sys-fs\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.667951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667937 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-agent-certs\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:01:31.667997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.667985 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-netns\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.668040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668008 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-dbus\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.668040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668034 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.668100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668073 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.668131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668106 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:31.668179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-cnibin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.668218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668201 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-agent-certs\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:01:31.668350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668328 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-system-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.668419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668373 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.668419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668384 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:31.668539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668420 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.668539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668426 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649bz\" (UniqueName: \"kubernetes.io/projected/17053989-3da4-4c70-92c3-949cd0b733ee-kube-api-access-649bz\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.668539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668470 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17053989-3da4-4c70-92c3-949cd0b733ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.668539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668478 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-env-overrides\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668581 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c8451-1aac-44aa-a685-28b03801682e-ovn-node-metrics-cert\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668622 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbz8z\" (UniqueName: \"kubernetes.io/projected/0ed58f76-099a-498d-844d-0069904b507b-kube-api-access-qbz8z\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668643 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-env-overrides\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668656 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-tmp\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668659 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668700 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-host\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668731 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-tmp\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.668750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668727 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-konnectivity-ca\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668778 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbwh\" (UniqueName: \"kubernetes.io/projected/1679fbb3-a322-4c15-9837-df1040cc3f01-kube-api-access-hgbwh\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fwf9\" (UniqueName: \"kubernetes.io/projected/24c63ae0-ce91-4f48-9690-45033ad525cb-kube-api-access-5fwf9\") pod \"migrator-66df5d496b-qtrbj\" (UID: \"24c63ae0-ce91-4f48-9690-45033ad525cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668840 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-log-socket\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668866 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd55j\" (UniqueName: \"kubernetes.io/projected/849c8451-1aac-44aa-a685-28b03801682e-kube-api-access-hd55j\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668892 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-conf-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668893 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/849c8451-1aac-44aa-a685-28b03801682e-ovn-node-metrics-cert\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.669140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.668953 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-bin\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.669462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669415 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.669543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669463 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66b75f9f-c202-4a6c-b370-c5ce3d1de74c-konnectivity-ca\") pod \"konnectivity-agent-cg74r\" (UID: \"66b75f9f-c202-4a6c-b370-c5ce3d1de74c\") " pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:01:31.669598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669545 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed58f76-099a-498d-844d-0069904b507b-cert\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:01:31.669649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669600 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-trusted-ca\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.669701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669645 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.669873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669854 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-run\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.669941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669877 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17053989-3da4-4c70-92c3-949cd0b733ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.669941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669918 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.670024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669955 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.670024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669981 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t4t9\" (UniqueName: \"kubernetes.io/projected/2df37093-9c54-4d41-ac8b-b41d8f83579d-kube-api-access-6t4t9\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.670024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669995 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed58f76-099a-498d-844d-0069904b507b-cert\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:01:31.670024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.669983 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-trusted-ca\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.670024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670016 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.670240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670185 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.670240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670221 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.670335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670255 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:31.670335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670270 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.670335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670285 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-socket-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.670464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670340 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-kubelet\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.670464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670376 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-stats-auth\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.670464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670408 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-multus-certs\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.670464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670448 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qz2s\" (UniqueName: \"kubernetes.io/projected/a5c72c64-5332-4820-870d-8a38037b2604-kube-api-access-6qz2s\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.670730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670710 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b7ef6fe-524b-4542-b281-27f7fce2deb8-original-pull-secret\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.670970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670945 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:31.671088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.670997 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.671154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671109 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.671226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671204 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-stats-auth\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.671280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671210 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-config\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.671346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671273 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17053989-3da4-4c70-92c3-949cd0b733ee-crio-socket\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.671459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671343 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76vc\" (UniqueName: \"kubernetes.io/projected/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-kube-api-access-f76vc\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.671459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671395 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/333ef57f-4887-4c8f-81c8-962265ad3be0-config\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.671459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671429 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.671643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671472 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.671643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671586 5815 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:31.671643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671606 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9kl\" (UniqueName: \"kubernetes.io/projected/837adf32-c68a-46ef-bca6-5b5364f74576-kube-api-access-2z9kl\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.671791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671644 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-daemon-config\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.671791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671681 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/333ef57f-4887-4c8f-81c8-962265ad3be0-serving-cert\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.671791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671709 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-systemd\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.671791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671735 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757e39d9-7832-46de-8c68-930c0b80943e-metrics-certs\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:31.671791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671760 5815 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:31.671791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671767 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-socket-dir-parent\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.672050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671818 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-kubelet\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.672050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671857 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4faac008-58e3-41b1-9f56-8a8760071532-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:31.672050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671887 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.672050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671906 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-node-log\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.672050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.671991 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-config\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.672050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672011 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.672324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672203 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/333ef57f-4887-4c8f-81c8-962265ad3be0-serving-cert\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672349 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672437 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672480 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-registration-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672533 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-netd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672546 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-daemon-config\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672563 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.672945 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-metrics-certs\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673054 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-config\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673180 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673186 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673259 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-var-lib-kubelet\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673313 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysconfig\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673368 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4faac008-58e3-41b1-9f56-8a8760071532-nginx-conf\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673379 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtlv\" (UniqueName: \"kubernetes.io/projected/757e39d9-7832-46de-8c68-930c0b80943e-kube-api-access-wqtlv\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673464 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4wn\" (UniqueName: \"kubernetes.io/projected/3da65b23-4da0-4d6e-8475-0a5bc3c19768-kube-api-access-2h4wn\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673493 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673562 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673638 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-multus\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673702 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673777 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xqn\" (UniqueName: \"kubernetes.io/projected/8ba183e8-f0a3-41fa-9247-20576448a07f-kube-api-access-r5xqn\") pod \"migrator-66df5d496b-xmd4t\" (UID: \"8ba183e8-f0a3-41fa-9247-20576448a07f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673813 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-etc-selinux\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673840 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-netns\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673878 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfhg\" (UniqueName: \"kubernetes.io/projected/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-kube-api-access-9bfhg\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673940 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-host\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673969 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcm6\" (UniqueName: \"kubernetes.io/projected/60d3aaa0-ccf0-42ae-92d5-44512e33aed4-kube-api-access-pjcm6\") pod \"volume-data-source-validator-58c9f5f46d-4gnrh\" (UID: \"60d3aaa0-ccf0-42ae-92d5-44512e33aed4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.673996 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674008 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674026 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57sb\" (UniqueName: \"kubernetes.io/projected/333ef57f-4887-4c8f-81c8-962265ad3be0-kube-api-access-j57sb\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674060 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-system-cni-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674103 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-etc-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674343 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674544 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674583 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674631 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674664 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-sys\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674704 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-var-lib-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674733 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82de9774-f285-46a2-a1d0-cf66b6f780b2-hosts-file\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674776 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-tmp-dir\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674806 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvpb\" (UniqueName: \"kubernetes.io/projected/05ab659c-a2dc-4764-a537-4b86fba538d2-kube-api-access-6mvpb\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674843 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674873 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674902 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82de9774-f285-46a2-a1d0-cf66b6f780b2-tmp-dir\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674931 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-serviceca\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674959 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-kubernetes\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.674992 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-slash\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675019 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-ovn\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675042 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675072 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-default-certificate\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675102 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-k8s-cni-cncf-io\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675155 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-hostroot\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675185 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-kubelet-config\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675216 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675246 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-modprobe-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675276 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-tuned\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675299 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-device-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675328 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-systemd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675358 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17053989-3da4-4c70-92c3-949cd0b733ee-data-volume\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675386 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675413 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-conf\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675437 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675627 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-tuned\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675698 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da65b23-4da0-4d6e-8475-0a5bc3c19768-iptables-alerter-script\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675721 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-tmp-dir\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675757 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da65b23-4da0-4d6e-8475-0a5bc3c19768-host-slash\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675795 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhsm\" (UniqueName: \"kubernetes.io/projected/82de9774-f285-46a2-a1d0-cf66b6f780b2-kube-api-access-blhsm\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675815 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675818 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-metrics-tls\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17053989-3da4-4c70-92c3-949cd0b733ee-data-volume\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676062 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676062 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676123 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4faac008-58e3-41b1-9f56-8a8760071532-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-zwgdc\" (UID: \"4faac008-58e3-41b1-9f56-8a8760071532\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-zwgdc" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676202 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82de9774-f285-46a2-a1d0-cf66b6f780b2-tmp-dir\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676251 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837adf32-c68a-46ef-bca6-5b5364f74576-service-ca-bundle\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676304 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/837adf32-c68a-46ef-bca6-5b5364f74576-default-certificate\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.675830 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-config-volume\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676352 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5c72c64-5332-4820-870d-8a38037b2604-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676367 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4fl\" (UniqueName: \"kubernetes.io/projected/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-kube-api-access-zp4fl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676420 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-cni-binary-copy\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676459 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-bin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.676533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676565 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676625 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-cnibin\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676641 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1679fbb3-a322-4c15-9837-df1040cc3f01-cni-binary-copy\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676704 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3da65b23-4da0-4d6e-8475-0a5bc3c19768-iptables-alerter-script\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676736 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-systemd-units\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:31.676766 5815 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676787 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-script-lib\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:31.676841 5815 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-81.us-east-2.compute.internal" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676881 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-serviceca\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.677085 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/849c8451-1aac-44aa-a685-28b03801682e-ovnkube-script-lib\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.676489 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-config-volume\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.678242 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kckq\" (UniqueName: \"kubernetes.io/projected/02c8db60-a588-40e9-a3b7-ba1ac37a5d47-kube-api-access-7kckq\") pod \"cluster-samples-operator-6ddb6c8b94-mbgph\" (UID: \"02c8db60-a588-40e9-a3b7-ba1ac37a5d47\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-mbgph" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.678923 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-649bz\" (UniqueName: \"kubernetes.io/projected/17053989-3da4-4c70-92c3-949cd0b733ee-kube-api-access-649bz\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.679434 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fwf9\" (UniqueName: \"kubernetes.io/projected/24c63ae0-ce91-4f48-9690-45033ad525cb-kube-api-access-5fwf9\") pod \"migrator-66df5d496b-qtrbj\" (UID: \"24c63ae0-ce91-4f48-9690-45033ad525cb\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qtrbj" Nov 03 20:01:31.679750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.679631 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbwh\" (UniqueName: \"kubernetes.io/projected/1679fbb3-a322-4c15-9837-df1040cc3f01-kube-api-access-hgbwh\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.681171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.681150 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") pod \"periodic-gathering-g9msr-f58ll\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:31.681239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.681197 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t4t9\" (UniqueName: \"kubernetes.io/projected/2df37093-9c54-4d41-ac8b-b41d8f83579d-kube-api-access-6t4t9\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.684580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.684565 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.685839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.685824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbz8z\" (UniqueName: \"kubernetes.io/projected/0ed58f76-099a-498d-844d-0069904b507b-kube-api-access-qbz8z\") pod \"ingress-canary-mt748\" (UID: \"0ed58f76-099a-498d-844d-0069904b507b\") " pod="openshift-ingress-canary/ingress-canary-mt748" Nov 03 20:01:31.687715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.687700 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd55j\" (UniqueName: \"kubernetes.io/projected/849c8451-1aac-44aa-a685-28b03801682e-kube-api-access-hd55j\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.693157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.693137 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4fl\" (UniqueName: \"kubernetes.io/projected/714f1a1a-a854-4a9c-a2f6-d07990fe9f0e-kube-api-access-zp4fl\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2\" (UID: \"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" Nov 03 20:01:31.693555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.693536 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xqn\" (UniqueName: \"kubernetes.io/projected/8ba183e8-f0a3-41fa-9247-20576448a07f-kube-api-access-r5xqn\") pod \"migrator-66df5d496b-xmd4t\" (UID: \"8ba183e8-f0a3-41fa-9247-20576448a07f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xmd4t" Nov 03 20:01:31.693555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.693548 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qz2s\" (UniqueName: \"kubernetes.io/projected/a5c72c64-5332-4820-870d-8a38037b2604-kube-api-access-6qz2s\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.693621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.693541 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnl8s\" (UniqueName: \"kubernetes.io/projected/7978e0e4-9979-42cf-a15b-767a03352919-kube-api-access-nnl8s\") pod \"network-check-target-kglzd\" (UID: \"7978e0e4-9979-42cf-a15b-767a03352919\") " pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:31.695778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.695749 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4wn\" (UniqueName: \"kubernetes.io/projected/3da65b23-4da0-4d6e-8475-0a5bc3c19768-kube-api-access-2h4wn\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.695976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.695961 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtlv\" (UniqueName: \"kubernetes.io/projected/757e39d9-7832-46de-8c68-930c0b80943e-kube-api-access-wqtlv\") pod \"network-metrics-daemon-bpqrl\" (UID: \"757e39d9-7832-46de-8c68-930c0b80943e\") " pod="openshift-multus/network-metrics-daemon-bpqrl" Nov 03 20:01:31.696025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.696000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57sb\" (UniqueName: \"kubernetes.io/projected/333ef57f-4887-4c8f-81c8-962265ad3be0-kube-api-access-j57sb\") pod \"console-operator-7cd56b8dbc-mxp5k\" (UID: \"333ef57f-4887-4c8f-81c8-962265ad3be0\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:31.698371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.698356 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfhg\" (UniqueName: \"kubernetes.io/projected/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-kube-api-access-9bfhg\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.698492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.698470 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhsm\" (UniqueName: \"kubernetes.io/projected/82de9774-f285-46a2-a1d0-cf66b6f780b2-kube-api-access-blhsm\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.698735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.698716 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") pod \"image-registry-5d87675b69-6m79x\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:31.698867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.698851 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcm6\" (UniqueName: \"kubernetes.io/projected/60d3aaa0-ccf0-42ae-92d5-44512e33aed4-kube-api-access-pjcm6\") pod \"volume-data-source-validator-58c9f5f46d-4gnrh\" (UID: \"60d3aaa0-ccf0-42ae-92d5-44512e33aed4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4gnrh" Nov 03 20:01:31.700626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.700610 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9kl\" (UniqueName: \"kubernetes.io/projected/837adf32-c68a-46ef-bca6-5b5364f74576-kube-api-access-2z9kl\") pod \"router-default-585db4bb4c-xs84v\" (UID: \"837adf32-c68a-46ef-bca6-5b5364f74576\") " pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:31.700889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.700869 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76vc\" (UniqueName: \"kubernetes.io/projected/99e197f7-de69-4fb1-b6d8-e1ffaa6d4903-kube-api-access-f76vc\") pod \"dns-default-d6zdd\" (UID: \"99e197f7-de69-4fb1-b6d8-e1ffaa6d4903\") " pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:31.701013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.700997 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvpb\" (UniqueName: \"kubernetes.io/projected/05ab659c-a2dc-4764-a537-4b86fba538d2-kube-api-access-6mvpb\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778007 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-cnibin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778033 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-system-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778050 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-host\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.778129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778065 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-log-socket\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778117 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75f38eb4-b4e8-4673-a540-24c8ebe84ee8-host\") pod \"node-ca-n27x8\" (UID: \"75f38eb4-b4e8-4673-a540-24c8ebe84ee8\") " pod="openshift-image-registry/node-ca-n27x8" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778118 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-conf-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778143 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-system-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778156 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-bin\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778175 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-conf-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778187 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-run\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778146 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-cnibin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778188 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-bin\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778175 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-log-socket\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778224 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778238 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-run\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778268 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-socket-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778286 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-kubelet\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778307 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-multus-certs\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778317 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778329 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17053989-3da4-4c70-92c3-949cd0b733ee-crio-socket\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778332 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-socket-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778345 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778365 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-systemd\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778395 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-socket-dir-parent\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778400 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778405 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17053989-3da4-4c70-92c3-949cd0b733ee-crio-socket\") pod \"insights-runtime-extractor-9p8gl\" (UID: \"17053989-3da4-4c70-92c3-949cd0b733ee\") " pod="openshift-insights/insights-runtime-extractor-9p8gl" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778343 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-kubelet\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778358 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-multus-certs\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778410 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-kubelet\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778441 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-socket-dir-parent\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778444 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-kubelet\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778442 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-systemd\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778455 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-node-log\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778485 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-node-log\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778489 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778533 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-registration-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778577 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-registration-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778577 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778593 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-netd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778625 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-var-lib-kubelet\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778628 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-cni-netd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778645 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysconfig\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778685 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysconfig\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778678 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-var-lib-kubelet\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778689 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778724 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-multus\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778745 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-multus-cni-dir\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778765 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-etc-selinux\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778789 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-netns\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778813 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-host\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778829 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-etc-selinux\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778790 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-multus\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778837 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-system-cni-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778859 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-etc-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778864 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-netns\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778880 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-sys\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.778865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778891 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-host\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778899 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-var-lib-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778907 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-system-cni-dir\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778912 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-etc-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778929 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-var-lib-openvswitch\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778927 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82de9774-f285-46a2-a1d0-cf66b6f780b2-hosts-file\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778950 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-sys\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778951 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-kubernetes\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778965 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82de9774-f285-46a2-a1d0-cf66b6f780b2-hosts-file\") pod \"node-resolver-j4p9c\" (UID: \"82de9774-f285-46a2-a1d0-cf66b6f780b2\") " pod="openshift-dns/node-resolver-j4p9c" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.778994 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-slash\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-kubernetes\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779015 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-ovn\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779052 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779077 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-k8s-cni-cncf-io\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779090 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-ovn\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779096 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-slash\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-hostroot\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779138 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-kubelet-config\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779134 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-hostroot\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779151 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779165 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-modprobe-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779190 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-device-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779201 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-kubelet-config\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779218 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-systemd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779245 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779266 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-conf\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779270 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-run-systemd\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779246 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-device-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779287 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779299 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-modprobe-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779185 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-k8s-cni-cncf-io\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779311 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da65b23-4da0-4d6e-8475-0a5bc3c19768-host-slash\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779338 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-bin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779359 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-cnibin\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779382 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-systemd-units\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779385 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3da65b23-4da0-4d6e-8475-0a5bc3c19768-host-slash\") pod \"iptables-alerter-q4r6s\" (UID: \"3da65b23-4da0-4d6e-8475-0a5bc3c19768\") " pod="openshift-network-operator/iptables-alerter-q4r6s" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779408 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-lib-modules\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779419 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-conf\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-etc-kubernetes\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779438 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779453 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-sys-fs\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779458 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-var-lib-cni-bin\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779477 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-netns\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779490 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5c72c64-5332-4820-870d-8a38037b2604-cnibin\") pod \"multus-additional-cni-plugins-kvv5k\" (UID: \"a5c72c64-5332-4820-870d-8a38037b2604\") " pod="openshift-multus/multus-additional-cni-plugins-kvv5k" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779501 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-dbus\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779535 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-lib-modules\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/849c8451-1aac-44aa-a685-28b03801682e-systemd-units\") pod \"ovnkube-node-dt8zf\" (UID: \"849c8451-1aac-44aa-a685-28b03801682e\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779361 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05ab659c-a2dc-4764-a537-4b86fba538d2-etc-sysctl-d\") pod \"tuned-htdgg\" (UID: \"05ab659c-a2dc-4764-a537-4b86fba538d2\") " pod="openshift-cluster-node-tuning-operator/tuned-htdgg" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779609 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-host-run-netns\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779646 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1679fbb3-a322-4c15-9837-df1040cc3f01-etc-kubernetes\") pod \"multus-88mp6\" (UID: \"1679fbb3-a322-4c15-9837-df1040cc3f01\") " pod="openshift-multus/multus-88mp6" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779668 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2df37093-9c54-4d41-ac8b-b41d8f83579d-sys-fs\") pod \"aws-ebs-csi-driver-node-stbpw\" (UID: \"2df37093-9c54-4d41-ac8b-b41d8f83579d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-stbpw" Nov 03 20:01:31.780192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.779706 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b7ef6fe-524b-4542-b281-27f7fce2deb8-dbus\") pod \"global-pull-secret-syncer-6k5bk\" (UID: \"3b7ef6fe-524b-4542-b281-27f7fce2deb8\") " pod="kube-system/global-pull-secret-syncer-6k5bk" Nov 03 20:01:31.850196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.850132 5815 scope.go:122] "RemoveContainer" containerID="c0952f28baf936512f566d16da0236fd83f8112e3a0beebedfa0c585a9628b48" Nov 03 20:01:31.851046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:31.850756 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:32.659791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:32.659764 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:01:32.660266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:32.659917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" event={"ID":"333ef57f-4887-4c8f-81c8-962265ad3be0","Type":"ContainerStarted","Data":"3501ea03dac3a62ac0f61b880038474987bcf87905fa37bbf6e5b0f118e1a803"} Nov 03 20:01:32.852273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:32.852246 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:33.391070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:33.391036 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:33.664535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:33.664462 5815 generic.go:356] "Generic (PLEG): container finished" podID="f07a2feb-3648-4f19-9385-baeb5eeeaeeb" containerID="27d9102d81a8eae6b6102c3c76ea548adcea5725476d04366079f88e31103ab3" exitCode=0 Nov 03 20:01:33.664849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:33.664552 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hft4t" event={"ID":"f07a2feb-3648-4f19-9385-baeb5eeeaeeb","Type":"ContainerDied","Data":"27d9102d81a8eae6b6102c3c76ea548adcea5725476d04366079f88e31103ab3"} Nov 03 20:01:33.664972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:33.664958 5815 scope.go:122] "RemoveContainer" containerID="27d9102d81a8eae6b6102c3c76ea548adcea5725476d04366079f88e31103ab3" Nov 03 20:01:33.752145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:33.752120 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:34.254845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.254810 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cf8vr-g6gm2"] Nov 03 20:01:34.272711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.272686 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cf8vr-g6gm2"] Nov 03 20:01:34.272854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.272775 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.403006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.402981 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.403118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.403026 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.403118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.403047 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgfx\" (UniqueName: \"kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.403187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.403162 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.504152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.504133 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.504239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.504162 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgfx\" (UniqueName: \"kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.504239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.504183 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.504239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.504234 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.504554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.504532 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.504875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.504840 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.505057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.505039 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.514827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.514797 5815 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:01:34.516928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.516906 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgfx\" (UniqueName: \"kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx\") pod \"periodic-gathering-cf8vr-g6gm2\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.599303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.599286 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:01:34.671979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.671946 5815 generic.go:356] "Generic (PLEG): container finished" podID="714f1a1a-a854-4a9c-a2f6-d07990fe9f0e" containerID="9799885ea4c0085789344f3e6493f558cf3321afaf5af14f18ecfc114ee922a8" exitCode=0 Nov 03 20:01:34.672383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.672052 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" event={"ID":"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e","Type":"ContainerDied","Data":"9799885ea4c0085789344f3e6493f558cf3321afaf5af14f18ecfc114ee922a8"} Nov 03 20:01:34.672383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.672343 5815 scope.go:122] "RemoveContainer" containerID="9799885ea4c0085789344f3e6493f558cf3321afaf5af14f18ecfc114ee922a8" Nov 03 20:01:34.675380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.675351 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hft4t" event={"ID":"f07a2feb-3648-4f19-9385-baeb5eeeaeeb","Type":"ContainerStarted","Data":"724b8125f3d7c1c232fde55b1278b3802abd98c10b4ffc3fbdfcc422fa9fc2e2"} Nov 03 20:01:34.884165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:34.884093 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cf8vr-g6gm2"] Nov 03 20:01:34.887871 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:34.887839 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f7b1e8_8c25_4dbe_b4ff_e9c78d744b22.slice/crio-cc518f7df55f988cd006bf2c6425135f9458b80ba0465188939f18b23e1ffdab WatchSource:0}: Error finding container cc518f7df55f988cd006bf2c6425135f9458b80ba0465188939f18b23e1ffdab: Status 404 returned error can't find the container with id cc518f7df55f988cd006bf2c6425135f9458b80ba0465188939f18b23e1ffdab Nov 03 20:01:35.680966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:35.680929 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" event={"ID":"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22","Type":"ContainerStarted","Data":"c5b60d54767e23891876ecbb3e37f51f7227583ca356480477e54045659c9c93"} Nov 03 20:01:35.681425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:35.680982 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" event={"ID":"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22","Type":"ContainerStarted","Data":"cc518f7df55f988cd006bf2c6425135f9458b80ba0465188939f18b23e1ffdab"} Nov 03 20:01:35.682997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:35.682972 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fk6v2" event={"ID":"714f1a1a-a854-4a9c-a2f6-d07990fe9f0e","Type":"ContainerStarted","Data":"48b6893360d3c04e2b21d8422ebbccfe3cdd9547ffb19a74d3a7b6682e36f3e8"} Nov 03 20:01:35.702753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:35.702709 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" podStartSLOduration=1.702695329 podStartE2EDuration="1.702695329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:35.702145147 +0000 UTC m=+5.232490819" watchObservedRunningTime="2026-11-03 20:01:35.702695329 +0000 UTC m=+5.233040998" Nov 03 20:01:36.152757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.152727 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6zdd_99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/dns/0.log" Nov 03 20:01:36.323906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.323884 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6zdd_99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/kube-rbac-proxy/0.log" Nov 03 20:01:36.327038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.327013 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:01:36.327234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.327222 5815 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:01:36.689278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.689248 5815 generic.go:356] "Generic (PLEG): container finished" podID="8370c5cf-f442-4e7d-b8ab-ea6ede18552b" containerID="de5c36f1e723eeda1251f6019b60209c45a2b0df8c9fbc63b998dcbb6fd29305" exitCode=0 Nov 03 20:01:36.689732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.689318 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" event={"ID":"8370c5cf-f442-4e7d-b8ab-ea6ede18552b","Type":"ContainerDied","Data":"de5c36f1e723eeda1251f6019b60209c45a2b0df8c9fbc63b998dcbb6fd29305"} Nov 03 20:01:36.689816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.689800 5815 scope.go:122] "RemoveContainer" containerID="de5c36f1e723eeda1251f6019b60209c45a2b0df8c9fbc63b998dcbb6fd29305" Nov 03 20:01:36.861392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:36.861358 5815 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cg74r_66b75f9f-c202-4a6c-b370-c5ce3d1de74c/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:37.013608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.013580 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-smkqz"] Nov 03 20:01:37.016008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.015985 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.040467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.040447 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:01:37.040945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.040919 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:01:37.041108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.041050 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ls9s\"" Nov 03 20:01:37.084882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.084855 5815 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cg74r_66b75f9f-c202-4a6c-b370-c5ce3d1de74c/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:37.095363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.095336 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:01:37.095363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.095336 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:01:37.123447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123415 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-wtmp\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123450 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-sys\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123485 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-root\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123605 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-textfile\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123633 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-tls\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123724 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-metrics-client-ca\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123751 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbzj9\" (UniqueName: \"kubernetes.io/projected/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-kube-api-access-bbzj9\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123863 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.123917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.123899 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-accelerators-collector-config\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.124131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.124107 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j4p9c_82de9774-f285-46a2-a1d0-cf66b6f780b2/dns-node-resolver/0.log" Nov 03 20:01:37.224694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.224654 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-metrics-client-ca\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.224694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.224687 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbzj9\" (UniqueName: \"kubernetes.io/projected/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-kube-api-access-bbzj9\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.224933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.224904 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.224964 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-accelerators-collector-config\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225049 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-wtmp\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225087 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-sys\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225123 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-root\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225160 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-textfile\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225186 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-tls\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225334 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-sys\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225402 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-metrics-client-ca\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225473 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-root\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225495 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-wtmp\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225580 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-accelerators-collector-config\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.225946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.225919 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-textfile\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.227470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.227451 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.227781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.227762 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-node-exporter-tls\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.234995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.234970 5815 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cg74r_66b75f9f-c202-4a6c-b370-c5ce3d1de74c/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:37.236631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.236615 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbzj9\" (UniqueName: \"kubernetes.io/projected/4783c1cb-2a65-4ab1-822a-e54f6d9ba570-kube-api-access-bbzj9\") pod \"node-exporter-smkqz\" (UID: \"4783c1cb-2a65-4ab1-822a-e54f6d9ba570\") " pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.311207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.311114 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:01:37.334790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.334764 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-smkqz" Nov 03 20:01:37.347832 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:01:37.347803 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4783c1cb_2a65_4ab1_822a_e54f6d9ba570.slice/crio-2d924e2048bb54f645dee8c789429c2201f2b56cd1ea827edd9d8b2a3fa79cef WatchSource:0}: Error finding container 2d924e2048bb54f645dee8c789429c2201f2b56cd1ea827edd9d8b2a3fa79cef: Status 404 returned error can't find the container with id 2d924e2048bb54f645dee8c789429c2201f2b56cd1ea827edd9d8b2a3fa79cef Nov 03 20:01:37.350078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.350060 5815 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:01:37.386294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.386266 5815 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cg74r_66b75f9f-c202-4a6c-b370-c5ce3d1de74c/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:37.528445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.528414 5815 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cg74r_66b75f9f-c202-4a6c-b370-c5ce3d1de74c/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:37.528623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.528605 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-585db4bb4c-xs84v_837adf32-c68a-46ef-bca6-5b5364f74576/router/0.log" Nov 03 20:01:37.695043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.694976 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-j5jf2" event={"ID":"8370c5cf-f442-4e7d-b8ab-ea6ede18552b","Type":"ContainerStarted","Data":"e9f203cf4bf069d794201908f8f57128ecb0a6319e0674c09a104d86fbe48d44"} Nov 03 20:01:37.696157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.696134 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-smkqz" event={"ID":"4783c1cb-2a65-4ab1-822a-e54f6d9ba570","Type":"ContainerStarted","Data":"2d924e2048bb54f645dee8c789429c2201f2b56cd1ea827edd9d8b2a3fa79cef"} Nov 03 20:01:37.927559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:37.927500 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mt748_0ed58f76-099a-498d-844d-0069904b507b/serve-healthcheck-canary/0.log" Nov 03 20:01:38.329034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.328998 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-mbgph_02c8db60-a588-40e9-a3b7-ba1ac37a5d47/cluster-samples-operator/0.log" Nov 03 20:01:38.439577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.439550 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:38.439707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.439676 5815 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:01:38.443989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.443966 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mxp5k" Nov 03 20:01:38.538459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.538395 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-mbgph_02c8db60-a588-40e9-a3b7-ba1ac37a5d47/cluster-samples-operator-watch/0.log" Nov 03 20:01:38.701646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.701564 5815 generic.go:356] "Generic (PLEG): container finished" podID="4783c1cb-2a65-4ab1-822a-e54f6d9ba570" containerID="7d75375710bc66a6e3adf60632826536df5e792aae5f2bb093c894c60cd0e7e8" exitCode=0 Nov 03 20:01:38.702046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:38.701663 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-smkqz" event={"ID":"4783c1cb-2a65-4ab1-822a-e54f6d9ba570","Type":"ContainerDied","Data":"7d75375710bc66a6e3adf60632826536df5e792aae5f2bb093c894c60cd0e7e8"} Nov 03 20:01:39.393069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:39.393036 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d6zdd" Nov 03 20:01:39.708109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:39.708031 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-smkqz" event={"ID":"4783c1cb-2a65-4ab1-822a-e54f6d9ba570","Type":"ContainerStarted","Data":"68f102f0dc43be381e1a9866cd1b85066dd5ece9fad2dea16f5d2cf4990fc4ff"} Nov 03 20:01:39.708109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:39.708098 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-smkqz" event={"ID":"4783c1cb-2a65-4ab1-822a-e54f6d9ba570","Type":"ContainerStarted","Data":"7eb5dd5673b4ce133f37181a498f0e74fcf877ccdb233620686bd65af7c26f2c"} Nov 03 20:01:39.736805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:39.736757 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-smkqz" podStartSLOduration=2.712098086 podStartE2EDuration="3.736743848s" totalImagesPullingTime="947.910831ms" totalInitContainerRuntime="76.734931ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:39.735683994 +0000 UTC m=+9.266029684" watchObservedRunningTime="2026-11-03 20:01:39.736743848 +0000 UTC m=+9.267089514" Nov 03 20:01:40.481298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:40.481265 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:40.481465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:40.481415 5815 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:01:40.482590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:40.482573 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-585db4bb4c-xs84v" Nov 03 20:01:41.591896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:41.591866 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:41.595900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:41.595876 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:01:43.172402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:43.172370 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d87675b69-6m79x"] Nov 03 20:01:43.849982 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:43.849946 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:01:53.737633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:53.737593 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-g9msr-f58ll"] Nov 03 20:01:53.738283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:53.737797 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-g9msr-f58ll" podUID="033c2d08-c440-495d-9957-0b7d41914e1a" containerName="insights-gathering" containerID="cri-o://65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a" gracePeriod=30 Nov 03 20:01:53.982766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:53.982744 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:54.053862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.053786 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") pod \"033c2d08-c440-495d-9957-0b7d41914e1a\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " Nov 03 20:01:54.053862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.053838 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") pod \"033c2d08-c440-495d-9957-0b7d41914e1a\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " Nov 03 20:01:54.054044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.053868 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") pod \"033c2d08-c440-495d-9957-0b7d41914e1a\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " Nov 03 20:01:54.054044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.053916 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") pod \"033c2d08-c440-495d-9957-0b7d41914e1a\" (UID: \"033c2d08-c440-495d-9957-0b7d41914e1a\") " Nov 03 20:01:54.054166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.054132 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path" pod "033c2d08-c440-495d-9957-0b7d41914e1a" (UID: "033c2d08-c440-495d-9957-0b7d41914e1a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:01:54.054276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.054247 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle" pod "033c2d08-c440-495d-9957-0b7d41914e1a" (UID: "033c2d08-c440-495d-9957-0b7d41914e1a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:54.054355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.054315 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle" pod "033c2d08-c440-495d-9957-0b7d41914e1a" (UID: "033c2d08-c440-495d-9957-0b7d41914e1a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:01:54.056256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.056235 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn" pod "033c2d08-c440-495d-9957-0b7d41914e1a" (UID: "033c2d08-c440-495d-9957-0b7d41914e1a"). InnerVolumeSpecName "kube-api-access-r7nbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:01:54.103929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.103907 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cg74r" Nov 03 20:01:54.154595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.154564 5815 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-service-ca-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:54.154595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.154595 5815 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/033c2d08-c440-495d-9957-0b7d41914e1a-trusted-ca-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:54.154706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.154605 5815 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/033c2d08-c440-495d-9957-0b7d41914e1a-archives-path\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:54.154706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.154615 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nbn\" (UniqueName: \"kubernetes.io/projected/033c2d08-c440-495d-9957-0b7d41914e1a-kube-api-access-r7nbn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:01:54.768809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.768779 5815 generic.go:356] "Generic (PLEG): container finished" podID="033c2d08-c440-495d-9957-0b7d41914e1a" containerID="65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a" exitCode=2 Nov 03 20:01:54.769283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.768838 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-g9msr-f58ll" Nov 03 20:01:54.769283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.768862 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9msr-f58ll" event={"ID":"033c2d08-c440-495d-9957-0b7d41914e1a","Type":"ContainerDied","Data":"65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a"} Nov 03 20:01:54.769283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.768926 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-g9msr-f58ll" event={"ID":"033c2d08-c440-495d-9957-0b7d41914e1a","Type":"ContainerDied","Data":"b70e4806378420d1d3df40ad0527d778056e8ab0215c737188bca437385c97ff"} Nov 03 20:01:54.769283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.768949 5815 scope.go:122] "RemoveContainer" containerID="65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a" Nov 03 20:01:54.778785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.778767 5815 scope.go:122] "RemoveContainer" containerID="65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a" Nov 03 20:01:54.779083 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:01:54.779064 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a\": container with ID starting with 65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a not found: ID does not exist" containerID="65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a" Nov 03 20:01:54.779136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.779092 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a"} err="failed to get container status \"65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a\": rpc error: code = NotFound desc = could not find container \"65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a\": container with ID starting with 65a2429d7c831281099351e4fd0097a50c4ad8e35da8068065b678cb7ec66c4a not found: ID does not exist" Nov 03 20:01:54.785748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.785729 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-g9msr-f58ll"] Nov 03 20:01:54.791359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:54.791340 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-g9msr-f58ll"] Nov 03 20:01:56.583670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:56.583642 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033c2d08-c440-495d-9957-0b7d41914e1a" path="/var/lib/kubelet/pods/033c2d08-c440-495d-9957-0b7d41914e1a/volumes" Nov 03 20:01:57.323493 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:01:57.323469 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kglzd" Nov 03 20:02:00.682394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:00.682365 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt8zf" Nov 03 20:02:08.193154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.193097 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" podUID="b843f245-431e-40af-a566-a4fa62de8c39" containerName="registry" containerID="cri-o://6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0" gracePeriod=30 Nov 03 20:02:08.437062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.437040 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:02:08.572839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.572807 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.572893 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.572918 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.572943 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.572970 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.572993 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.573041 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.573094 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") pod \"b843f245-431e-40af-a566-a4fa62de8c39\" (UID: \"b843f245-431e-40af-a566-a4fa62de8c39\") " Nov 03 20:02:08.573645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.573613 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:08.573808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.573756 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:08.575887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.575854 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "kube-api-access-qxn75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:08.576051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.576023 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:08.576264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.576240 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:08.576377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.576289 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:08.576377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.576340 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:08.579726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.579699 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted" pod "b843f245-431e-40af-a566-a4fa62de8c39" (UID: "b843f245-431e-40af-a566-a4fa62de8c39"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:08.673758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673729 5815 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-image-registry-private-configuration\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673752 5815 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b843f245-431e-40af-a566-a4fa62de8c39-installation-pull-secrets\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673764 5815 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b843f245-431e-40af-a566-a4fa62de8c39-ca-trust-extracted\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673773 5815 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-registry-certificates\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673781 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn75\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-kube-api-access-qxn75\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673790 5815 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-registry-tls\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673798 5815 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b843f245-431e-40af-a566-a4fa62de8c39-bound-sa-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.673943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.673806 5815 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b843f245-431e-40af-a566-a4fa62de8c39-trusted-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:08.827281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.827216 5815 generic.go:356] "Generic (PLEG): container finished" podID="b843f245-431e-40af-a566-a4fa62de8c39" containerID="6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0" exitCode=0 Nov 03 20:02:08.827281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.827275 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" Nov 03 20:02:08.827443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.827298 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" event={"ID":"b843f245-431e-40af-a566-a4fa62de8c39","Type":"ContainerDied","Data":"6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0"} Nov 03 20:02:08.827443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.827348 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d87675b69-6m79x" event={"ID":"b843f245-431e-40af-a566-a4fa62de8c39","Type":"ContainerDied","Data":"6bb36c71cef7423cee2370fd4a08b5a0eff8f95e2eb56fdbdc7a6029cf5e5336"} Nov 03 20:02:08.827443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.827368 5815 scope.go:122] "RemoveContainer" containerID="6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0" Nov 03 20:02:08.835812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.835798 5815 scope.go:122] "RemoveContainer" containerID="6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0" Nov 03 20:02:08.836055 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:02:08.836039 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0\": container with ID starting with 6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0 not found: ID does not exist" containerID="6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0" Nov 03 20:02:08.836118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.836061 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0"} err="failed to get container status \"6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0\": rpc error: code = NotFound desc = could not find container \"6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0\": container with ID starting with 6b67e4694541ec2a9189feaad054a8b3ab6dbfbac0878212613b2145c90d5ab0 not found: ID does not exist" Nov 03 20:02:08.844448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.844430 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d87675b69-6m79x"] Nov 03 20:02:08.848695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:08.848676 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d87675b69-6m79x"] Nov 03 20:02:10.583141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:02:10.583111 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b843f245-431e-40af-a566-a4fa62de8c39" path="/var/lib/kubelet/pods/b843f245-431e-40af-a566-a4fa62de8c39/volumes" Nov 03 20:02:13.849300 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:02:13.849268 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:02:43.850038 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:02:43.850002 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:03:02.015059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:02.015027 5815 generic.go:356] "Generic (PLEG): container finished" podID="25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" containerID="c5b60d54767e23891876ecbb3e37f51f7227583ca356480477e54045659c9c93" exitCode=0 Nov 03 20:03:02.015485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:02.015080 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" event={"ID":"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22","Type":"ContainerDied","Data":"c5b60d54767e23891876ecbb3e37f51f7227583ca356480477e54045659c9c93"} Nov 03 20:03:03.150602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.150580 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:03:03.252075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252048 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path\") pod \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " Nov 03 20:03:03.252187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252136 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle\") pod \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " Nov 03 20:03:03.252227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252206 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle\") pod \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " Nov 03 20:03:03.252261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252237 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx\" (UniqueName: \"kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx\") pod \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\" (UID: \"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22\") " Nov 03 20:03:03.252365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252339 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path" pod "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" (UID: "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:03.252495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252478 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle" pod "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" (UID: "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:03.252557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252539 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle" pod "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" (UID: "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:03:03.252619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252605 5815 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-archives-path\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.252657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252621 5815 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-trusted-ca-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.252657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.252630 5815 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-service-ca-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:03.254462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.254434 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx" pod "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" (UID: "25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22"). InnerVolumeSpecName "kube-api-access-9rgfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:03.353860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:03.353812 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgfx\" (UniqueName: \"kubernetes.io/projected/25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22-kube-api-access-9rgfx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:04.023703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:04.023666 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" event={"ID":"25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22","Type":"ContainerDied","Data":"cc518f7df55f988cd006bf2c6425135f9458b80ba0465188939f18b23e1ffdab"} Nov 03 20:03:04.023703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:04.023702 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cf8vr-g6gm2" Nov 03 20:03:04.023900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:03:04.023715 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc518f7df55f988cd006bf2c6425135f9458b80ba0465188939f18b23e1ffdab" Nov 03 20:03:13.849750 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:03:13.849673 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:03:43.851052 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:03:43.851013 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:04:13.850224 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:04:13.850188 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:04:43.850892 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:04:43.850806 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:05:13.851010 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:05:13.850971 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:05:43.850538 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:05:43.850474 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:05:52.063946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.063912 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4"] Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064248 5815 cpu_manager.go:414] "containerMap: removing container" podUID="033c2d08-c440-495d-9957-0b7d41914e1a" containerName="insights-gathering" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064257 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033c2d08-c440-495d-9957-0b7d41914e1a" containerName="insights-gathering" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064288 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b843f245-431e-40af-a566-a4fa62de8c39" containerName="registry" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064292 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b843f245-431e-40af-a566-a4fa62de8c39" containerName="registry" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064313 5815 cpu_manager.go:414] "containerMap: removing container" podUID="25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" containerName="insights-gathering" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064320 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" containerName="insights-gathering" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064372 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="033c2d08-c440-495d-9957-0b7d41914e1a" containerName="insights-gathering" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064383 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b843f245-431e-40af-a566-a4fa62de8c39" containerName="registry" Nov 03 20:05:52.064466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.064390 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22" containerName="insights-gathering" Nov 03 20:05:52.067585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.067568 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.072497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.072479 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:05:52.072497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.072485 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:05:52.072642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.072500 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lvqck\"" Nov 03 20:05:52.076607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.076583 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4"] Nov 03 20:05:52.181472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.181437 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.181651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.181490 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnmd\" (UniqueName: \"kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.181651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.181606 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.206449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.206424 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh"] Nov 03 20:05:52.209920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.209906 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.214018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.213995 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-wx85f\"" Nov 03 20:05:52.219908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.219886 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh"] Nov 03 20:05:52.279583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.279560 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf"] Nov 03 20:05:52.282915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.282893 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnmd\" (UniqueName: \"kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.283028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.282919 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.283028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.282959 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.283028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.282959 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.283196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.283068 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.283196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.283097 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nw78\" (UniqueName: \"kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.283196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.283122 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.283365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.283284 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.283420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.283401 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.286473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.286453 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-w95l5\"" Nov 03 20:05:52.292527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.292493 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf"] Nov 03 20:05:52.294860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.294838 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnmd\" (UniqueName: \"kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.376603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.376535 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt"] Nov 03 20:05:52.379871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.379858 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.383326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383300 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:05:52.383645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383621 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.383711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383668 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.383711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383632 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-pgjdp\"" Nov 03 20:05:52.383788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383738 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvmq\" (UniqueName: \"kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.383827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383781 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nw78\" (UniqueName: \"kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.383827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383813 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.383900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383836 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.383993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.383974 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.384116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.384097 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.389142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.389123 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt"] Nov 03 20:05:52.397280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.397261 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nw78\" (UniqueName: \"kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.485079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485052 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.485211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485089 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.485211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485112 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvmq\" (UniqueName: \"kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.485286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485231 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.485393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485375 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.485452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485440 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9n7\" (UniqueName: \"kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.485543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485524 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.485586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.485547 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.498029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.498005 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvmq\" (UniqueName: \"kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.526011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.525982 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:05:52.586347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.586321 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.586484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.586375 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9n7\" (UniqueName: \"kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.586484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.586399 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.586858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.586839 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.586942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.586853 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.598839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.598815 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:05:52.599156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.599135 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9n7\" (UniqueName: \"kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.664573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.664368 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4"] Nov 03 20:05:52.666808 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:52.666776 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0110869_0337_4f2c_a3df_0970c03daa99.slice/crio-eecc7a066ff5e4c2b42fce43346a2052228c9ab7775cb33762138031e37a6994 WatchSource:0}: Error finding container eecc7a066ff5e4c2b42fce43346a2052228c9ab7775cb33762138031e37a6994: Status 404 returned error can't find the container with id eecc7a066ff5e4c2b42fce43346a2052228c9ab7775cb33762138031e37a6994 Nov 03 20:05:52.700581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.700558 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:05:52.798576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.798555 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh"] Nov 03 20:05:52.800773 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:52.800745 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9eca1ad_ce84_4886_8c10_0ffc84e48d81.slice/crio-3e2f6bd65df83b943971b8cf39bc83cb57e3ab835d2ca79bdb6951634711c389 WatchSource:0}: Error finding container 3e2f6bd65df83b943971b8cf39bc83cb57e3ab835d2ca79bdb6951634711c389: Status 404 returned error can't find the container with id 3e2f6bd65df83b943971b8cf39bc83cb57e3ab835d2ca79bdb6951634711c389 Nov 03 20:05:52.869824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.869798 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf"] Nov 03 20:05:52.873085 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:52.873052 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fe32ee_8cef_4e52_b598_0557aac91cf5.slice/crio-bbbc7ba06c268da42647a26554767afe1b385416842153fc69eaeaa0ce325e3b WatchSource:0}: Error finding container bbbc7ba06c268da42647a26554767afe1b385416842153fc69eaeaa0ce325e3b: Status 404 returned error can't find the container with id bbbc7ba06c268da42647a26554767afe1b385416842153fc69eaeaa0ce325e3b Nov 03 20:05:52.971632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:52.971610 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt"] Nov 03 20:05:52.974541 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:52.974498 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02146de4_34ad_4c05_8608_e65b7d7a5e2e.slice/crio-8aecda9412545f6b8874127550f6db5eadba7ec7c9a3441e01e6c00b81d80a3e WatchSource:0}: Error finding container 8aecda9412545f6b8874127550f6db5eadba7ec7c9a3441e01e6c00b81d80a3e: Status 404 returned error can't find the container with id 8aecda9412545f6b8874127550f6db5eadba7ec7c9a3441e01e6c00b81d80a3e Nov 03 20:05:53.053864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.053842 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s"] Nov 03 20:05:53.058849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.058833 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.062625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.062605 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xmgv5\"" Nov 03 20:05:53.067181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.067156 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s"] Nov 03 20:05:53.192011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.191977 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.192189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.192032 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrd4n\" (UniqueName: \"kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.192260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.192190 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.293499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.293403 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.293499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.293493 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.293499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.293536 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrd4n\" (UniqueName: \"kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.294164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.294141 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.294431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.294331 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.307227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.307199 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrd4n\" (UniqueName: \"kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.377195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.377165 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:05:53.623909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.623758 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" event={"ID":"c8fe32ee-8cef-4e52-b598-0557aac91cf5","Type":"ContainerStarted","Data":"bbbc7ba06c268da42647a26554767afe1b385416842153fc69eaeaa0ce325e3b"} Nov 03 20:05:53.625954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.625870 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" event={"ID":"d9eca1ad-ce84-4886-8c10-0ffc84e48d81","Type":"ContainerStarted","Data":"3e2f6bd65df83b943971b8cf39bc83cb57e3ab835d2ca79bdb6951634711c389"} Nov 03 20:05:53.628119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.628086 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" event={"ID":"c0110869-0337-4f2c-a3df-0970c03daa99","Type":"ContainerStarted","Data":"eecc7a066ff5e4c2b42fce43346a2052228c9ab7775cb33762138031e37a6994"} Nov 03 20:05:53.630396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.630367 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" event={"ID":"02146de4-34ad-4c05-8608-e65b7d7a5e2e","Type":"ContainerStarted","Data":"8aecda9412545f6b8874127550f6db5eadba7ec7c9a3441e01e6c00b81d80a3e"} Nov 03 20:05:53.665150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.665124 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s"] Nov 03 20:05:53.668964 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:53.668940 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced1bef7_136a_4c76_91b8_6e54b41d216a.slice/crio-8bb929c3411786eb223449a040eb643afb6e4902a0435eb9199b8b43856e9cc0 WatchSource:0}: Error finding container 8bb929c3411786eb223449a040eb643afb6e4902a0435eb9199b8b43856e9cc0: Status 404 returned error can't find the container with id 8bb929c3411786eb223449a040eb643afb6e4902a0435eb9199b8b43856e9cc0 Nov 03 20:05:53.879117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.879035 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm"] Nov 03 20:05:53.883938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.883916 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:53.888086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.888064 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-c4gh5\"" Nov 03 20:05:53.892227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:53.892149 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm"] Nov 03 20:05:54.001768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.001732 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.001928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.001816 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvb4f\" (UniqueName: \"kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.001928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.001862 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.102780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.102751 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvb4f\" (UniqueName: \"kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.103199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.102804 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.103199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.102849 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.103199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.103143 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.103927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.103631 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.118189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.118140 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvb4f\" (UniqueName: \"kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.207983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.207677 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:05:54.516160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.516000 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm"] Nov 03 20:05:54.643007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.642914 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" event={"ID":"73159259-793c-4f6e-bd3f-734776bb5d41","Type":"ContainerStarted","Data":"a7dcad58987144849ff7a4eeb863bdc1682b1dc62c4e2b6146484600a24fc4a3"} Nov 03 20:05:54.646576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:54.646467 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" event={"ID":"ced1bef7-136a-4c76-91b8-6e54b41d216a","Type":"ContainerStarted","Data":"8bb929c3411786eb223449a040eb643afb6e4902a0435eb9199b8b43856e9cc0"} Nov 03 20:05:56.691935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.690131 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx"] Nov 03 20:05:56.694894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.694068 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.701067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.701012 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-zj5gs\"" Nov 03 20:05:56.704054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.704037 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx"] Nov 03 20:05:56.828825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.828785 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hg9\" (UniqueName: \"kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.829020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.828899 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.829020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.828943 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.930094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.930053 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hg9\" (UniqueName: \"kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.930275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.930169 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.930275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.930208 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.930647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.930626 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.931214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.931187 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:56.966257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:56.965542 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hg9\" (UniqueName: \"kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:57.017431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.017217 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:05:57.346620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.346589 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx"] Nov 03 20:05:57.667185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.667044 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" event={"ID":"c8302f87-54a4-411a-ab7a-cb84919e78b0","Type":"ContainerStarted","Data":"8e93a0382093b622ce02f1e7e7792906d2b511de416a3b6aefa38183dbe65bf4"} Nov 03 20:05:57.883979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.882370 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6"] Nov 03 20:05:57.886394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.885960 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:57.896551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.895873 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:05:57.896551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.896059 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-225mv\"" Nov 03 20:05:57.900355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.900326 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6"] Nov 03 20:05:57.974594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.972960 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt"] Nov 03 20:05:57.979768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.979104 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:57.990901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.990874 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt"] Nov 03 20:05:57.996941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:57.996916 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-89bbd\"" Nov 03 20:05:58.042188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042157 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042201 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042245 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjcr\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042278 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042321 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042354 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042383 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042420 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.042569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.042498 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.143818 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.143879 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjcr\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.143914 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.143942 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.143975 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.143999 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144035 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144073 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144101 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144133 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144161 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npnq9\" (UniqueName: \"kubernetes.io/projected/e6cb547b-57ea-4b55-9d70-6597fa2073a7-kube-api-access-npnq9\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144196 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144248 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144277 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144315 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144348 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144386 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.144410 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.145175 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.146179 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.146481 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.147309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.146969 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.149645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.149497 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.150131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.150002 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.150131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.150075 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.165562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.165471 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.166390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.166299 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjcr\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr\") pod \"maas-default-gateway-openshift-default-59f957fdd7-j9gl6\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.209635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.209350 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245426 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245487 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245541 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245565 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245622 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npnq9\" (UniqueName: \"kubernetes.io/projected/e6cb547b-57ea-4b55-9d70-6597fa2073a7-kube-api-access-npnq9\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245655 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245704 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245734 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.245769 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.246131 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.247041 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.247292 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.247534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.247450 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.248256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.247932 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.249430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.249369 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.251202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.251179 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.266253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.266203 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6cb547b-57ea-4b55-9d70-6597fa2073a7-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.268299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.268257 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npnq9\" (UniqueName: \"kubernetes.io/projected/e6cb547b-57ea-4b55-9d70-6597fa2073a7-kube-api-access-npnq9\") pod \"openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt\" (UID: \"e6cb547b-57ea-4b55-9d70-6597fa2073a7\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.300701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.300672 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:05:58.531162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.531082 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6"] Nov 03 20:05:58.539939 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:58.539910 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369568fe_f4a1_4dd5_904a_eb30dae71013.slice/crio-17b3db4a69913fa3ad56cdb4be00a8d0ef5e4c349afb8fefa0ec3b39f6f6d4e2 WatchSource:0}: Error finding container 17b3db4a69913fa3ad56cdb4be00a8d0ef5e4c349afb8fefa0ec3b39f6f6d4e2: Status 404 returned error can't find the container with id 17b3db4a69913fa3ad56cdb4be00a8d0ef5e4c349afb8fefa0ec3b39f6f6d4e2 Nov 03 20:05:58.621478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.621437 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt"] Nov 03 20:05:58.640667 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:05:58.640640 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cb547b_57ea_4b55_9d70_6597fa2073a7.slice/crio-29f156e3a77d09e08a9f45cab3e999754db6dd4c902d3ae267f59816b502297e WatchSource:0}: Error finding container 29f156e3a77d09e08a9f45cab3e999754db6dd4c902d3ae267f59816b502297e: Status 404 returned error can't find the container with id 29f156e3a77d09e08a9f45cab3e999754db6dd4c902d3ae267f59816b502297e Nov 03 20:05:58.680564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.680448 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" event={"ID":"369568fe-f4a1-4dd5-904a-eb30dae71013","Type":"ContainerStarted","Data":"17b3db4a69913fa3ad56cdb4be00a8d0ef5e4c349afb8fefa0ec3b39f6f6d4e2"} Nov 03 20:05:58.683065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:58.683035 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" event={"ID":"e6cb547b-57ea-4b55-9d70-6597fa2073a7","Type":"ContainerStarted","Data":"29f156e3a77d09e08a9f45cab3e999754db6dd4c902d3ae267f59816b502297e"} Nov 03 20:05:59.815001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.814965 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp"] Nov 03 20:05:59.819823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.819796 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:05:59.834391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.834358 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp"] Nov 03 20:05:59.845147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.845122 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln"] Nov 03 20:05:59.853581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.853556 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:05:59.858100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.858057 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln"] Nov 03 20:05:59.871681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.871656 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg"] Nov 03 20:05:59.876548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.876529 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:05:59.889906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.889878 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg"] Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964356 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964425 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964458 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964498 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88fqt\" (UniqueName: \"kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964547 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5spk\" (UniqueName: \"kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964584 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964607 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zln\" (UniqueName: \"kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964641 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:05:59.964876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:05:59.964663 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.065807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065733 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.065807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065769 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zln\" (UniqueName: \"kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.065807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065795 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.066090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065813 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.066090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065856 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.066090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065894 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.066090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.066090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065935 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88fqt\" (UniqueName: \"kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.066090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.065952 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5spk\" (UniqueName: \"kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.067517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.066561 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.067517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.066864 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.067517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.066890 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.067517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.067048 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.067517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.067204 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.067517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.067233 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.089881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.089833 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5spk\" (UniqueName: \"kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.090645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.090553 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88fqt\" (UniqueName: \"kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.094912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.094888 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zln\" (UniqueName: \"kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.175946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.175916 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:00.184228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.184028 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:00.198264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.198056 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:00.489391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.489361 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp"] Nov 03 20:06:00.509714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.509563 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg"] Nov 03 20:06:00.514237 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:06:00.514197 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode985dc3e_82c3_413a_b2a1_643ec3b7fd88.slice/crio-a3826728123ed6f908e47a0a17c1cd65bead9fe8aaa3d3e35ffa24cd80e4109a WatchSource:0}: Error finding container a3826728123ed6f908e47a0a17c1cd65bead9fe8aaa3d3e35ffa24cd80e4109a: Status 404 returned error can't find the container with id a3826728123ed6f908e47a0a17c1cd65bead9fe8aaa3d3e35ffa24cd80e4109a Nov 03 20:06:00.518601 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:06:00.518482 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a77c2b2_aeaa_4a16_a973_ec4baf851eec.slice/crio-e39767b23afcb36fcd7199c83fcd62702da580883ba8ce9d2fd17b1a6c1dca41 WatchSource:0}: Error finding container e39767b23afcb36fcd7199c83fcd62702da580883ba8ce9d2fd17b1a6c1dca41: Status 404 returned error can't find the container with id e39767b23afcb36fcd7199c83fcd62702da580883ba8ce9d2fd17b1a6c1dca41 Nov 03 20:06:00.538410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.538359 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln"] Nov 03 20:06:00.709167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.709106 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" event={"ID":"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3","Type":"ContainerStarted","Data":"5e6434c56232df28751b10b682954fbe63f76402f79e840ee51de696f45d7488"} Nov 03 20:06:00.716290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.716255 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" event={"ID":"e985dc3e-82c3-413a-b2a1-643ec3b7fd88","Type":"ContainerStarted","Data":"a3826728123ed6f908e47a0a17c1cd65bead9fe8aaa3d3e35ffa24cd80e4109a"} Nov 03 20:06:00.718631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:00.718578 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" event={"ID":"9a77c2b2-aeaa-4a16-a973-ec4baf851eec","Type":"ContainerStarted","Data":"e39767b23afcb36fcd7199c83fcd62702da580883ba8ce9d2fd17b1a6c1dca41"} Nov 03 20:06:10.055086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.054605 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:06:10.055086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.054696 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:06:10.055086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.054747 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:06:10.843742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.843575 5815 generic.go:356] "Generic (PLEG): container finished" podID="c0110869-0337-4f2c-a3df-0970c03daa99" containerID="df48715905e8b68541937735d0b87aa49db73ee52a947924ada36dc90fdfad5a" exitCode=0 Nov 03 20:06:10.843742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.843672 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" event={"ID":"c0110869-0337-4f2c-a3df-0970c03daa99","Type":"ContainerDied","Data":"df48715905e8b68541937735d0b87aa49db73ee52a947924ada36dc90fdfad5a"} Nov 03 20:06:10.851431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.850723 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" event={"ID":"369568fe-f4a1-4dd5-904a-eb30dae71013","Type":"ContainerStarted","Data":"b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347"} Nov 03 20:06:10.855903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.855182 5815 generic.go:356] "Generic (PLEG): container finished" podID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerID="5c2bb74a48d6303e79ba4436667f84ffb773ab5cc032b67956977d7ee3da679e" exitCode=0 Nov 03 20:06:10.855903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.855297 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" event={"ID":"02146de4-34ad-4c05-8608-e65b7d7a5e2e","Type":"ContainerDied","Data":"5c2bb74a48d6303e79ba4436667f84ffb773ab5cc032b67956977d7ee3da679e"} Nov 03 20:06:10.888622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:10.887667 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podStartSLOduration=2.376084005 podStartE2EDuration="13.887648959s" totalImagesPullingTime="11.511564954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:10.886699448 +0000 UTC m=+280.417045124" watchObservedRunningTime="2026-11-03 20:06:10.887648959 +0000 UTC m=+280.417994632" Nov 03 20:06:11.210861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:11.210789 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:06:11.217721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:11.217455 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:06:11.866546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:11.866471 5815 generic.go:356] "Generic (PLEG): container finished" podID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerID="34357bf7e5e50deb86d7169a1c85d471c9f00a7f8f39a106bbbac84da363dbbf" exitCode=0 Nov 03 20:06:11.867617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:11.867587 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" event={"ID":"c8fe32ee-8cef-4e52-b598-0557aac91cf5","Type":"ContainerDied","Data":"34357bf7e5e50deb86d7169a1c85d471c9f00a7f8f39a106bbbac84da363dbbf"} Nov 03 20:06:11.867786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:11.867645 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:06:11.868833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:11.868773 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:06:12.873324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:12.873282 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" event={"ID":"ced1bef7-136a-4c76-91b8-6e54b41d216a","Type":"ContainerStarted","Data":"e81d4e37c4bdbe92c135c028eb06240501ea0e0e9f16b4db1d1e15c6941601ef"} Nov 03 20:06:12.875695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:12.875665 5815 generic.go:356] "Generic (PLEG): container finished" podID="c0110869-0337-4f2c-a3df-0970c03daa99" containerID="a5a232557f9ba9f69dc0021715c5f3ee51a74048a0fd270375afef46d3ec446a" exitCode=0 Nov 03 20:06:12.875827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:12.875756 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" event={"ID":"c0110869-0337-4f2c-a3df-0970c03daa99","Type":"ContainerDied","Data":"a5a232557f9ba9f69dc0021715c5f3ee51a74048a0fd270375afef46d3ec446a"} Nov 03 20:06:13.858346 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:06:13.858281 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:06:13.883397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.883365 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerID="0f459b58480284f529f0132bcf16847112f4fe27603542a9c5b60566263f05cb" exitCode=0 Nov 03 20:06:13.883914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.883410 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" event={"ID":"9a77c2b2-aeaa-4a16-a973-ec4baf851eec","Type":"ContainerDied","Data":"0f459b58480284f529f0132bcf16847112f4fe27603542a9c5b60566263f05cb"} Nov 03 20:06:13.887947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.886993 5815 generic.go:356] "Generic (PLEG): container finished" podID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerID="b195ada92b689bb6f3465fb5f06c0f5ee1972086ac7fcfa8243d3aa37af4d82a" exitCode=0 Nov 03 20:06:13.887947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.887698 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" event={"ID":"d9eca1ad-ce84-4886-8c10-0ffc84e48d81","Type":"ContainerDied","Data":"b195ada92b689bb6f3465fb5f06c0f5ee1972086ac7fcfa8243d3aa37af4d82a"} Nov 03 20:06:13.889964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.889936 5815 generic.go:356] "Generic (PLEG): container finished" podID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerID="e81d4e37c4bdbe92c135c028eb06240501ea0e0e9f16b4db1d1e15c6941601ef" exitCode=0 Nov 03 20:06:13.890066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.890045 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" event={"ID":"ced1bef7-136a-4c76-91b8-6e54b41d216a","Type":"ContainerDied","Data":"e81d4e37c4bdbe92c135c028eb06240501ea0e0e9f16b4db1d1e15c6941601ef"} Nov 03 20:06:13.891978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.891954 5815 generic.go:356] "Generic (PLEG): container finished" podID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerID="140b70933161fc251cd4137ce861451216c9c3afa8d21975051c8908fbe57811" exitCode=0 Nov 03 20:06:13.892146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.892121 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" event={"ID":"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3","Type":"ContainerDied","Data":"140b70933161fc251cd4137ce861451216c9c3afa8d21975051c8908fbe57811"} Nov 03 20:06:13.894248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.894226 5815 generic.go:356] "Generic (PLEG): container finished" podID="73159259-793c-4f6e-bd3f-734776bb5d41" containerID="ba5f552f79fce827bacbb2fa4f9d30a1e087efa574649d58faae12a2f04aaf05" exitCode=0 Nov 03 20:06:13.895197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:13.895176 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" event={"ID":"73159259-793c-4f6e-bd3f-734776bb5d41","Type":"ContainerDied","Data":"ba5f552f79fce827bacbb2fa4f9d30a1e087efa574649d58faae12a2f04aaf05"} Nov 03 20:06:14.368531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.368474 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:06:14.368629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.368571 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:06:14.368629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.368605 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:06:14.902658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.902569 5815 generic.go:356] "Generic (PLEG): container finished" podID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerID="711f99169ac1fa08f17710fa8aaa72d383eb455f42d03631b3e4b9a41f22b74f" exitCode=0 Nov 03 20:06:14.903077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.902695 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" event={"ID":"ced1bef7-136a-4c76-91b8-6e54b41d216a","Type":"ContainerDied","Data":"711f99169ac1fa08f17710fa8aaa72d383eb455f42d03631b3e4b9a41f22b74f"} Nov 03 20:06:14.905268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.905245 5815 generic.go:356] "Generic (PLEG): container finished" podID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerID="7cee44c1bec7786900a2dfc6a674a6077b2f964a4516eadbf80360454fb560ca" exitCode=0 Nov 03 20:06:14.905347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.905333 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" event={"ID":"c8302f87-54a4-411a-ab7a-cb84919e78b0","Type":"ContainerDied","Data":"7cee44c1bec7786900a2dfc6a674a6077b2f964a4516eadbf80360454fb560ca"} Nov 03 20:06:14.908018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.907998 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" event={"ID":"e6cb547b-57ea-4b55-9d70-6597fa2073a7","Type":"ContainerStarted","Data":"88567b33270c1aaa9dbea7daae5eb48302d8d9585ce6a1e4d724e90f02032647"} Nov 03 20:06:14.909638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.909611 5815 generic.go:356] "Generic (PLEG): container finished" podID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerID="962776ff5be31b1be2998744cc2c06850d2aa56bc80b79a265b58b4845204858" exitCode=0 Nov 03 20:06:14.909738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.909648 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" event={"ID":"e985dc3e-82c3-413a-b2a1-643ec3b7fd88","Type":"ContainerDied","Data":"962776ff5be31b1be2998744cc2c06850d2aa56bc80b79a265b58b4845204858"} Nov 03 20:06:14.911756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.911700 5815 generic.go:356] "Generic (PLEG): container finished" podID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerID="fc008ea458fc778a031cf9d7a5e290d7ee64b7a1eecda7d03b7a413b58536700" exitCode=0 Nov 03 20:06:14.911823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.911777 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" event={"ID":"02146de4-34ad-4c05-8608-e65b7d7a5e2e","Type":"ContainerDied","Data":"fc008ea458fc778a031cf9d7a5e290d7ee64b7a1eecda7d03b7a413b58536700"} Nov 03 20:06:14.914281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.914256 5815 generic.go:356] "Generic (PLEG): container finished" podID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerID="128df5a5a197d157a169ceeef4eedbf3313c7aef2a4e8b73554037928923f134" exitCode=0 Nov 03 20:06:14.914369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.914341 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" event={"ID":"c8fe32ee-8cef-4e52-b598-0557aac91cf5","Type":"ContainerDied","Data":"128df5a5a197d157a169ceeef4eedbf3313c7aef2a4e8b73554037928923f134"} Nov 03 20:06:14.922982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.922918 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerID="53ee244f2464b7eeb4fc8c02626a04e9244d8b0c6caa7ce003790c4e24fa2488" exitCode=0 Nov 03 20:06:14.923047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:14.923013 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" event={"ID":"9a77c2b2-aeaa-4a16-a973-ec4baf851eec","Type":"ContainerDied","Data":"53ee244f2464b7eeb4fc8c02626a04e9244d8b0c6caa7ce003790c4e24fa2488"} Nov 03 20:06:15.009293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.009251 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" podStartSLOduration=2.285907608 podStartE2EDuration="18.009238254s" totalImagesPullingTime="15.723330646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:15.00723654 +0000 UTC m=+284.537582232" watchObservedRunningTime="2026-11-03 20:06:15.009238254 +0000 UTC m=+284.539583960" Nov 03 20:06:15.301604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.301566 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:06:15.307161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.306935 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:06:15.934053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.933031 5815 generic.go:356] "Generic (PLEG): container finished" podID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerID="125de0cb045cdbb13d2a1cae3cfb80f9cc762a69da21693a3d4aeb700e280bb5" exitCode=0 Nov 03 20:06:15.934053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.933143 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" event={"ID":"d9eca1ad-ce84-4886-8c10-0ffc84e48d81","Type":"ContainerDied","Data":"125de0cb045cdbb13d2a1cae3cfb80f9cc762a69da21693a3d4aeb700e280bb5"} Nov 03 20:06:15.942580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.941723 5815 generic.go:356] "Generic (PLEG): container finished" podID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerID="d89c33569d24f31b3f541135b7ccd92f8a1e4594ac012cdef3ab351e84d733e5" exitCode=0 Nov 03 20:06:15.942580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.941818 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" event={"ID":"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3","Type":"ContainerDied","Data":"d89c33569d24f31b3f541135b7ccd92f8a1e4594ac012cdef3ab351e84d733e5"} Nov 03 20:06:15.951217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.949624 5815 generic.go:356] "Generic (PLEG): container finished" podID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerID="ca9e4ee0aa0321d7f326481e78e408e26e4f48476942a32cb43429d3d2caa5a9" exitCode=0 Nov 03 20:06:15.951217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.950917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" event={"ID":"e985dc3e-82c3-413a-b2a1-643ec3b7fd88","Type":"ContainerDied","Data":"ca9e4ee0aa0321d7f326481e78e408e26e4f48476942a32cb43429d3d2caa5a9"} Nov 03 20:06:15.951217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.950957 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:06:15.952373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:15.952115 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt" Nov 03 20:06:22.014107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:22.014074 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" event={"ID":"c8302f87-54a4-411a-ab7a-cb84919e78b0","Type":"ContainerStarted","Data":"8ae286dfa7a42b003006a8262701808c29691c82caf77ddfcc094becb89c5e6d"} Nov 03 20:06:23.039606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:23.039087 5815 generic.go:356] "Generic (PLEG): container finished" podID="73159259-793c-4f6e-bd3f-734776bb5d41" containerID="af2b4f0ccb629d165394af4195cd033cd695cc777d84ad79a2214f87cac573e2" exitCode=0 Nov 03 20:06:23.039606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:23.039291 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" event={"ID":"73159259-793c-4f6e-bd3f-734776bb5d41","Type":"ContainerDied","Data":"af2b4f0ccb629d165394af4195cd033cd695cc777d84ad79a2214f87cac573e2"} Nov 03 20:06:23.055947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:23.054428 5815 generic.go:356] "Generic (PLEG): container finished" podID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerID="8ae286dfa7a42b003006a8262701808c29691c82caf77ddfcc094becb89c5e6d" exitCode=0 Nov 03 20:06:23.055947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:23.054499 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" event={"ID":"c8302f87-54a4-411a-ab7a-cb84919e78b0","Type":"ContainerDied","Data":"8ae286dfa7a42b003006a8262701808c29691c82caf77ddfcc094becb89c5e6d"} Nov 03 20:06:30.682048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:30.682015 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:06:30.706047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:30.706022 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:06:30.707837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:30.707813 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:06:30.742664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:30.742640 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:06:32.182261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:32.181748 5815 generic.go:356] "Generic (PLEG): container finished" podID="c0110869-0337-4f2c-a3df-0970c03daa99" containerID="7222de5121b5d2395d4ecf58efa74e7f425aa88f82b7ade5f1a5c5a5f15799ee" exitCode=0 Nov 03 20:06:32.182261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:32.181828 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" event={"ID":"c0110869-0337-4f2c-a3df-0970c03daa99","Type":"ContainerDied","Data":"7222de5121b5d2395d4ecf58efa74e7f425aa88f82b7ade5f1a5c5a5f15799ee"} Nov 03 20:06:33.461320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.461004 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:06:33.537858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.537825 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util\") pod \"c0110869-0337-4f2c-a3df-0970c03daa99\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " Nov 03 20:06:33.538046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.537958 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle\") pod \"c0110869-0337-4f2c-a3df-0970c03daa99\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " Nov 03 20:06:33.538046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.538021 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd\" (UniqueName: \"kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd\") pod \"c0110869-0337-4f2c-a3df-0970c03daa99\" (UID: \"c0110869-0337-4f2c-a3df-0970c03daa99\") " Nov 03 20:06:33.539061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.539019 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle" pod "c0110869-0337-4f2c-a3df-0970c03daa99" (UID: "c0110869-0337-4f2c-a3df-0970c03daa99"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:33.544929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.544885 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util" pod "c0110869-0337-4f2c-a3df-0970c03daa99" (UID: "c0110869-0337-4f2c-a3df-0970c03daa99"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:33.552112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.552083 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd" pod "c0110869-0337-4f2c-a3df-0970c03daa99" (UID: "c0110869-0337-4f2c-a3df-0970c03daa99"). InnerVolumeSpecName "kube-api-access-jhnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:33.639879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.639844 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnmd\" (UniqueName: \"kubernetes.io/projected/c0110869-0337-4f2c-a3df-0970c03daa99-kube-api-access-jhnmd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:33.639879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.639881 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:33.640122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:33.639896 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0110869-0337-4f2c-a3df-0970c03daa99-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:34.228188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:34.228157 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" event={"ID":"c0110869-0337-4f2c-a3df-0970c03daa99","Type":"ContainerDied","Data":"eecc7a066ff5e4c2b42fce43346a2052228c9ab7775cb33762138031e37a6994"} Nov 03 20:06:34.228324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:34.228210 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eecc7a066ff5e4c2b42fce43346a2052228c9ab7775cb33762138031e37a6994" Nov 03 20:06:34.228324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:34.228308 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kjnw4" Nov 03 20:06:35.239958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:35.239926 5815 generic.go:356] "Generic (PLEG): container finished" podID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerID="6a1051fb8eb82648c45af30fef758fe413cb2098aa46017745aa2765ca263831" exitCode=0 Nov 03 20:06:35.240311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:35.240024 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" event={"ID":"c8fe32ee-8cef-4e52-b598-0557aac91cf5","Type":"ContainerDied","Data":"6a1051fb8eb82648c45af30fef758fe413cb2098aa46017745aa2765ca263831"} Nov 03 20:06:35.245344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:35.245320 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" event={"ID":"9a77c2b2-aeaa-4a16-a973-ec4baf851eec","Type":"ContainerStarted","Data":"efc46d649bec981606a2d8a0e97007f26ef2752e76c10c1c162337928a65f9f6"} Nov 03 20:06:35.283164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:35.283080 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" podStartSLOduration=2.456744567 podStartE2EDuration="36.283060918s" totalImagesPullingTime="33.826316351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:35.282554917 +0000 UTC m=+304.812900591" watchObservedRunningTime="2026-11-03 20:06:35.283060918 +0000 UTC m=+304.813406592" Nov 03 20:06:36.251876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.251848 5815 generic.go:356] "Generic (PLEG): container finished" podID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerID="48614731efb876fbe3f58f57159625cefa7172cb712c8f15da67e6a10188dd02" exitCode=0 Nov 03 20:06:36.252253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.251932 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" event={"ID":"02146de4-34ad-4c05-8608-e65b7d7a5e2e","Type":"ContainerDied","Data":"48614731efb876fbe3f58f57159625cefa7172cb712c8f15da67e6a10188dd02"} Nov 03 20:06:36.254063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.254043 5815 generic.go:356] "Generic (PLEG): container finished" podID="73159259-793c-4f6e-bd3f-734776bb5d41" containerID="c884c63e65ce4cc7856000c5c85a51a9cdbbd09bf16855ed0aedfa0ae63e0881" exitCode=0 Nov 03 20:06:36.254148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.254128 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" event={"ID":"73159259-793c-4f6e-bd3f-734776bb5d41","Type":"ContainerDied","Data":"c884c63e65ce4cc7856000c5c85a51a9cdbbd09bf16855ed0aedfa0ae63e0881"} Nov 03 20:06:36.256107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.256083 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerID="efc46d649bec981606a2d8a0e97007f26ef2752e76c10c1c162337928a65f9f6" exitCode=0 Nov 03 20:06:36.256199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.256143 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" event={"ID":"9a77c2b2-aeaa-4a16-a973-ec4baf851eec","Type":"ContainerDied","Data":"efc46d649bec981606a2d8a0e97007f26ef2752e76c10c1c162337928a65f9f6"} Nov 03 20:06:36.260723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.260704 5815 generic.go:356] "Generic (PLEG): container finished" podID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerID="1c0c8fb1080f4eced5bd1b7bc66a60ffb8d1d4273ceeb5697a7303e5ace82e3d" exitCode=0 Nov 03 20:06:36.260823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.260780 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" event={"ID":"ced1bef7-136a-4c76-91b8-6e54b41d216a","Type":"ContainerDied","Data":"1c0c8fb1080f4eced5bd1b7bc66a60ffb8d1d4273ceeb5697a7303e5ace82e3d"} Nov 03 20:06:36.436530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.436450 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:06:36.565528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.565483 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util\") pod \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " Nov 03 20:06:36.565677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.565553 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq\" (UniqueName: \"kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq\") pod \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " Nov 03 20:06:36.565677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.565599 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle\") pod \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\" (UID: \"c8fe32ee-8cef-4e52-b598-0557aac91cf5\") " Nov 03 20:06:36.566120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.566098 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle" pod "c8fe32ee-8cef-4e52-b598-0557aac91cf5" (UID: "c8fe32ee-8cef-4e52-b598-0557aac91cf5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:36.567723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.567700 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq" pod "c8fe32ee-8cef-4e52-b598-0557aac91cf5" (UID: "c8fe32ee-8cef-4e52-b598-0557aac91cf5"). InnerVolumeSpecName "kube-api-access-cxvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:36.571823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.571800 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util" pod "c8fe32ee-8cef-4e52-b598-0557aac91cf5" (UID: "c8fe32ee-8cef-4e52-b598-0557aac91cf5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:36.666757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.666734 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:36.666840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.666761 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8fe32ee-8cef-4e52-b598-0557aac91cf5-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:36.666840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:36.666771 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvmq\" (UniqueName: \"kubernetes.io/projected/c8fe32ee-8cef-4e52-b598-0557aac91cf5-kube-api-access-cxvmq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.267719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.267692 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" Nov 03 20:06:37.267719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.267698 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqndgf" event={"ID":"c8fe32ee-8cef-4e52-b598-0557aac91cf5","Type":"ContainerDied","Data":"bbbc7ba06c268da42647a26554767afe1b385416842153fc69eaeaa0ce325e3b"} Nov 03 20:06:37.268195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.267778 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbbc7ba06c268da42647a26554767afe1b385416842153fc69eaeaa0ce325e3b" Nov 03 20:06:37.271880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.271852 5815 generic.go:356] "Generic (PLEG): container finished" podID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerID="8e618339b1fcc0f85bb9f8de0a583e78671bbc1c1c77f31c829a7ba02d0ec307" exitCode=0 Nov 03 20:06:37.272786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.271959 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" event={"ID":"d9eca1ad-ce84-4886-8c10-0ffc84e48d81","Type":"ContainerDied","Data":"8e618339b1fcc0f85bb9f8de0a583e78671bbc1c1c77f31c829a7ba02d0ec307"} Nov 03 20:06:37.274496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.274475 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" event={"ID":"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3","Type":"ContainerStarted","Data":"fab5053f0979938c29bfbbdb93c7e7db9663e3c59b36f0b594c2ba61b206481f"} Nov 03 20:06:37.276620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.276596 5815 generic.go:356] "Generic (PLEG): container finished" podID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerID="c1072ce3f1eb535a77f94bb3bc1a9970bcc18580f64432f39d3f5d9c7682ba4b" exitCode=0 Nov 03 20:06:37.276733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.276624 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" event={"ID":"c8302f87-54a4-411a-ab7a-cb84919e78b0","Type":"ContainerDied","Data":"c1072ce3f1eb535a77f94bb3bc1a9970bcc18580f64432f39d3f5d9c7682ba4b"} Nov 03 20:06:37.278823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.278803 5815 generic.go:356] "Generic (PLEG): container finished" podID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerID="a662c3bf75a65746dd8141fa6fa4a97c8d896a6e4c015eee4b450d42de8a0861" exitCode=0 Nov 03 20:06:37.278929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.278917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" event={"ID":"e985dc3e-82c3-413a-b2a1-643ec3b7fd88","Type":"ContainerDied","Data":"a662c3bf75a65746dd8141fa6fa4a97c8d896a6e4c015eee4b450d42de8a0861"} Nov 03 20:06:37.375708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.375488 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" podStartSLOduration=3.254844561 podStartE2EDuration="38.375471169s" totalImagesPullingTime="35.120626608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:37.34985839 +0000 UTC m=+306.880204064" watchObservedRunningTime="2026-11-03 20:06:37.375471169 +0000 UTC m=+306.905816843" Nov 03 20:06:37.443687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.443665 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:37.481742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.481714 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:06:37.509426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.509400 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:06:37.544122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.544096 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:06:37.574974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.574952 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle\") pod \"73159259-793c-4f6e-bd3f-734776bb5d41\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " Nov 03 20:06:37.575131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575005 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle\") pod \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " Nov 03 20:06:37.575131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575054 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n\" (UniqueName: \"kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n\") pod \"ced1bef7-136a-4c76-91b8-6e54b41d216a\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " Nov 03 20:06:37.575131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575086 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util\") pod \"ced1bef7-136a-4c76-91b8-6e54b41d216a\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " Nov 03 20:06:37.575131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575114 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util\") pod \"73159259-793c-4f6e-bd3f-734776bb5d41\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575162 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f\" (UniqueName: \"kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f\") pod \"73159259-793c-4f6e-bd3f-734776bb5d41\" (UID: \"73159259-793c-4f6e-bd3f-734776bb5d41\") " Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575203 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle\") pod \"ced1bef7-136a-4c76-91b8-6e54b41d216a\" (UID: \"ced1bef7-136a-4c76-91b8-6e54b41d216a\") " Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575290 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util\") pod \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575328 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln\" (UniqueName: \"kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln\") pod \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\" (UID: \"9a77c2b2-aeaa-4a16-a973-ec4baf851eec\") " Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575400 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle" pod "73159259-793c-4f6e-bd3f-734776bb5d41" (UID: "73159259-793c-4f6e-bd3f-734776bb5d41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.575617 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.576260 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle" pod "9a77c2b2-aeaa-4a16-a973-ec4baf851eec" (UID: "9a77c2b2-aeaa-4a16-a973-ec4baf851eec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.576676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.576428 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle" pod "ced1bef7-136a-4c76-91b8-6e54b41d216a" (UID: "ced1bef7-136a-4c76-91b8-6e54b41d216a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.577934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.577909 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n" pod "ced1bef7-136a-4c76-91b8-6e54b41d216a" (UID: "ced1bef7-136a-4c76-91b8-6e54b41d216a"). InnerVolumeSpecName "kube-api-access-wrd4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:37.579283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.579257 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f" pod "73159259-793c-4f6e-bd3f-734776bb5d41" (UID: "73159259-793c-4f6e-bd3f-734776bb5d41"). InnerVolumeSpecName "kube-api-access-nvb4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:37.579470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.579450 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln" pod "9a77c2b2-aeaa-4a16-a973-ec4baf851eec" (UID: "9a77c2b2-aeaa-4a16-a973-ec4baf851eec"). InnerVolumeSpecName "kube-api-access-t9zln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:37.582777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.582758 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util" pod "ced1bef7-136a-4c76-91b8-6e54b41d216a" (UID: "ced1bef7-136a-4c76-91b8-6e54b41d216a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.584076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.584047 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util" pod "73159259-793c-4f6e-bd3f-734776bb5d41" (UID: "73159259-793c-4f6e-bd3f-734776bb5d41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.584190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.584175 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util" pod "9a77c2b2-aeaa-4a16-a973-ec4baf851eec" (UID: "9a77c2b2-aeaa-4a16-a973-ec4baf851eec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.676698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.676669 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util\") pod \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " Nov 03 20:06:37.676836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.676782 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7\" (UniqueName: \"kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7\") pod \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " Nov 03 20:06:37.676836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.676830 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle\") pod \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\" (UID: \"02146de4-34ad-4c05-8608-e65b7d7a5e2e\") " Nov 03 20:06:37.677028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677013 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zln\" (UniqueName: \"kubernetes.io/projected/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-kube-api-access-t9zln\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677035 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677050 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrd4n\" (UniqueName: \"kubernetes.io/projected/ced1bef7-136a-4c76-91b8-6e54b41d216a-kube-api-access-wrd4n\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677065 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677079 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73159259-793c-4f6e-bd3f-734776bb5d41-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677092 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvb4f\" (UniqueName: \"kubernetes.io/projected/73159259-793c-4f6e-bd3f-734776bb5d41-kube-api-access-nvb4f\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677104 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ced1bef7-136a-4c76-91b8-6e54b41d216a-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677117 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a77c2b2-aeaa-4a16-a973-ec4baf851eec-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.677330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.677269 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle" pod "02146de4-34ad-4c05-8608-e65b7d7a5e2e" (UID: "02146de4-34ad-4c05-8608-e65b7d7a5e2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.678942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.678912 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7" pod "02146de4-34ad-4c05-8608-e65b7d7a5e2e" (UID: "02146de4-34ad-4c05-8608-e65b7d7a5e2e"). InnerVolumeSpecName "kube-api-access-4t9n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:37.682013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.681990 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util" pod "02146de4-34ad-4c05-8608-e65b7d7a5e2e" (UID: "02146de4-34ad-4c05-8608-e65b7d7a5e2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:37.777682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.777637 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9n7\" (UniqueName: \"kubernetes.io/projected/02146de4-34ad-4c05-8608-e65b7d7a5e2e-kube-api-access-4t9n7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.777682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.777681 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:37.777682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:37.777692 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02146de4-34ad-4c05-8608-e65b7d7a5e2e-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.285711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.285674 5815 generic.go:356] "Generic (PLEG): container finished" podID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerID="fab5053f0979938c29bfbbdb93c7e7db9663e3c59b36f0b594c2ba61b206481f" exitCode=0 Nov 03 20:06:38.286166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.285763 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" event={"ID":"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3","Type":"ContainerDied","Data":"fab5053f0979938c29bfbbdb93c7e7db9663e3c59b36f0b594c2ba61b206481f"} Nov 03 20:06:38.287635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.287607 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" event={"ID":"02146de4-34ad-4c05-8608-e65b7d7a5e2e","Type":"ContainerDied","Data":"8aecda9412545f6b8874127550f6db5eadba7ec7c9a3441e01e6c00b81d80a3e"} Nov 03 20:06:38.287757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.287658 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aecda9412545f6b8874127550f6db5eadba7ec7c9a3441e01e6c00b81d80a3e" Nov 03 20:06:38.287757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.287616 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78dgpt" Nov 03 20:06:38.289649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.289624 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" event={"ID":"73159259-793c-4f6e-bd3f-734776bb5d41","Type":"ContainerDied","Data":"a7dcad58987144849ff7a4eeb863bdc1682b1dc62c4e2b6146484600a24fc4a3"} Nov 03 20:06:38.289858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.289663 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7dcad58987144849ff7a4eeb863bdc1682b1dc62c4e2b6146484600a24fc4a3" Nov 03 20:06:38.289858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.289633 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64429psm" Nov 03 20:06:38.291764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.291747 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" Nov 03 20:06:38.291764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.291755 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avj8kp" event={"ID":"9a77c2b2-aeaa-4a16-a973-ec4baf851eec","Type":"ContainerDied","Data":"e39767b23afcb36fcd7199c83fcd62702da580883ba8ce9d2fd17b1a6c1dca41"} Nov 03 20:06:38.291938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.291786 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e39767b23afcb36fcd7199c83fcd62702da580883ba8ce9d2fd17b1a6c1dca41" Nov 03 20:06:38.293894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.293874 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" Nov 03 20:06:38.293998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.293909 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718628s" event={"ID":"ced1bef7-136a-4c76-91b8-6e54b41d216a","Type":"ContainerDied","Data":"8bb929c3411786eb223449a040eb643afb6e4902a0435eb9199b8b43856e9cc0"} Nov 03 20:06:38.293998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.293946 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb929c3411786eb223449a040eb643afb6e4902a0435eb9199b8b43856e9cc0" Nov 03 20:06:38.444835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.444808 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:06:38.467684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.467658 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:38.477445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.477418 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:06:38.584322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584254 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78\" (UniqueName: \"kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78\") pod \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " Nov 03 20:06:38.584322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584287 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle\") pod \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " Nov 03 20:06:38.584322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util\") pod \"c8302f87-54a4-411a-ab7a-cb84919e78b0\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " Nov 03 20:06:38.584583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584341 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt\" (UniqueName: \"kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt\") pod \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " Nov 03 20:06:38.584583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584387 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util\") pod \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\" (UID: \"d9eca1ad-ce84-4886-8c10-0ffc84e48d81\") " Nov 03 20:06:38.584583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584419 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle\") pod \"c8302f87-54a4-411a-ab7a-cb84919e78b0\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " Nov 03 20:06:38.584583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584463 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle\") pod \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " Nov 03 20:06:38.584583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584527 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util\") pod \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\" (UID: \"e985dc3e-82c3-413a-b2a1-643ec3b7fd88\") " Nov 03 20:06:38.584583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.584558 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9\" (UniqueName: \"kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9\") pod \"c8302f87-54a4-411a-ab7a-cb84919e78b0\" (UID: \"c8302f87-54a4-411a-ab7a-cb84919e78b0\") " Nov 03 20:06:38.585247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.585129 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle" pod "d9eca1ad-ce84-4886-8c10-0ffc84e48d81" (UID: "d9eca1ad-ce84-4886-8c10-0ffc84e48d81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:38.585738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.585705 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle" pod "c8302f87-54a4-411a-ab7a-cb84919e78b0" (UID: "c8302f87-54a4-411a-ab7a-cb84919e78b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:38.585855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.585836 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle" pod "e985dc3e-82c3-413a-b2a1-643ec3b7fd88" (UID: "e985dc3e-82c3-413a-b2a1-643ec3b7fd88"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:38.586794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.586773 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt" pod "e985dc3e-82c3-413a-b2a1-643ec3b7fd88" (UID: "e985dc3e-82c3-413a-b2a1-643ec3b7fd88"). InnerVolumeSpecName "kube-api-access-88fqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:38.587183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.587164 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78" pod "d9eca1ad-ce84-4886-8c10-0ffc84e48d81" (UID: "d9eca1ad-ce84-4886-8c10-0ffc84e48d81"). InnerVolumeSpecName "kube-api-access-9nw78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:38.587267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.587236 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9" pod "c8302f87-54a4-411a-ab7a-cb84919e78b0" (UID: "c8302f87-54a4-411a-ab7a-cb84919e78b0"). InnerVolumeSpecName "kube-api-access-q9hg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:38.594275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.594252 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util" pod "e985dc3e-82c3-413a-b2a1-643ec3b7fd88" (UID: "e985dc3e-82c3-413a-b2a1-643ec3b7fd88"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:38.594534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.594495 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util" pod "d9eca1ad-ce84-4886-8c10-0ffc84e48d81" (UID: "d9eca1ad-ce84-4886-8c10-0ffc84e48d81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:38.594912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.594893 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util" pod "c8302f87-54a4-411a-ab7a-cb84919e78b0" (UID: "c8302f87-54a4-411a-ab7a-cb84919e78b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:38.686073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686045 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686068 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686077 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9hg9\" (UniqueName: \"kubernetes.io/projected/c8302f87-54a4-411a-ab7a-cb84919e78b0-kube-api-access-q9hg9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686087 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nw78\" (UniqueName: \"kubernetes.io/projected/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-kube-api-access-9nw78\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686096 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686105 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686113 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88fqt\" (UniqueName: \"kubernetes.io/projected/e985dc3e-82c3-413a-b2a1-643ec3b7fd88-kube-api-access-88fqt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686121 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9eca1ad-ce84-4886-8c10-0ffc84e48d81-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:38.686267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:38.686128 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8302f87-54a4-411a-ab7a-cb84919e78b0-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.300634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.300604 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" Nov 03 20:06:39.301067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.300600 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwxxfh" event={"ID":"d9eca1ad-ce84-4886-8c10-0ffc84e48d81","Type":"ContainerDied","Data":"3e2f6bd65df83b943971b8cf39bc83cb57e3ab835d2ca79bdb6951634711c389"} Nov 03 20:06:39.301067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.300752 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e2f6bd65df83b943971b8cf39bc83cb57e3ab835d2ca79bdb6951634711c389" Nov 03 20:06:39.302624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.302605 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" Nov 03 20:06:39.302748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.302620 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czzqqx" event={"ID":"c8302f87-54a4-411a-ab7a-cb84919e78b0","Type":"ContainerDied","Data":"8e93a0382093b622ce02f1e7e7792906d2b511de416a3b6aefa38183dbe65bf4"} Nov 03 20:06:39.302748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.302658 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e93a0382093b622ce02f1e7e7792906d2b511de416a3b6aefa38183dbe65bf4" Nov 03 20:06:39.304558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.304535 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" event={"ID":"e985dc3e-82c3-413a-b2a1-643ec3b7fd88","Type":"ContainerDied","Data":"a3826728123ed6f908e47a0a17c1cd65bead9fe8aaa3d3e35ffa24cd80e4109a"} Nov 03 20:06:39.304657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.304574 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3826728123ed6f908e47a0a17c1cd65bead9fe8aaa3d3e35ffa24cd80e4109a" Nov 03 20:06:39.304657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.304541 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sq8qg" Nov 03 20:06:39.433249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.433227 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:39.492215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.492186 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util\") pod \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " Nov 03 20:06:39.492347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.492247 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle\") pod \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " Nov 03 20:06:39.492385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.492362 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk\" (UniqueName: \"kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk\") pod \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\" (UID: \"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3\") " Nov 03 20:06:39.492789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.492763 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle" pod "ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" (UID: "ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.494520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.494479 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk" pod "ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" (UID: "ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3"). InnerVolumeSpecName "kube-api-access-t5spk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:39.497584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.497564 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util" pod "ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" (UID: "ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:39.593017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.592955 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5spk\" (UniqueName: \"kubernetes.io/projected/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-kube-api-access-t5spk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.593017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.592980 5815 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-util\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:39.593017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:39.592996 5815 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3-bundle\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:40.311044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:40.311011 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" event={"ID":"ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3","Type":"ContainerDied","Data":"5e6434c56232df28751b10b682954fbe63f76402f79e840ee51de696f45d7488"} Nov 03 20:06:40.311560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:40.311062 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e6434c56232df28751b10b682954fbe63f76402f79e840ee51de696f45d7488" Nov 03 20:06:40.311560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:40.311064 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chkwln" Nov 03 20:06:43.849567 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:06:43.849530 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:06:48.032285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032250 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq"] Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032580 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032592 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032604 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032610 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032618 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="extract" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032623 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="extract" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032634 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032639 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032644 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="extract" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032649 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="extract" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032656 5815 cpu_manager.go:414] "containerMap: removing container" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032660 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032666 5815 cpu_manager.go:414] "containerMap: removing container" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032670 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032677 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032681 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032689 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032692 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032698 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032702 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="pull" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032708 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032712 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="util" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032718 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="extract" Nov 03 20:06:48.032713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032725 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032731 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032735 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032741 5815 cpu_manager.go:414] "containerMap: removing container" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032745 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032751 5815 cpu_manager.go:414] "containerMap: removing container" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032768 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032775 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032780 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032785 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032789 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032797 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032801 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032806 5815 cpu_manager.go:414] "containerMap: removing container" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032810 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032817 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032821 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032825 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032829 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032836 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032840 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032847 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032852 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032858 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032861 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032869 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032873 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032878 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032882 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032888 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032891 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032897 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032901 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032906 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032911 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="pull" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032918 5815 cpu_manager.go:414] "containerMap: removing container" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032922 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="util" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032973 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8302f87-54a4-411a-ab7a-cb84919e78b0" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032983 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9eca1ad-ce84-4886-8c10-0ffc84e48d81" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032991 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced1bef7-136a-4c76-91b8-6e54b41d216a" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.032997 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a77c2b2-aeaa-4a16-a973-ec4baf851eec" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.033003 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="73159259-793c-4f6e-bd3f-734776bb5d41" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.033008 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e985dc3e-82c3-413a-b2a1-643ec3b7fd88" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.033013 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad634a0c-ff4d-4c2d-8896-6b0ac9b577d3" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.033018 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fe32ee-8cef-4e52-b598-0557aac91cf5" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.033022 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="02146de4-34ad-4c05-8608-e65b7d7a5e2e" containerName="extract" Nov 03 20:06:48.033480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.033028 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0110869-0337-4f2c-a3df-0970c03daa99" containerName="extract" Nov 03 20:06:48.035553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.035535 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.041069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.041044 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:06:48.041069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.041062 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:06:48.041211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.041121 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:06:48.041873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.041856 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-m6wx9\"" Nov 03 20:06:48.068872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.068852 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq"] Nov 03 20:06:48.121157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.121133 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:06:48.121312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.121196 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:06:48.159129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.159103 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.159244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.159137 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.159244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.159157 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.159244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.159179 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.159367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.159348 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.159403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.159380 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxm64\" (UniqueName: \"kubernetes.io/projected/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-kube-api-access-gxm64\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.259972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.259949 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.260107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.259980 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.260107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.260002 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.260107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.260031 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.260107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.260059 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxm64\" (UniqueName: \"kubernetes.io/projected/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-kube-api-access-gxm64\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.260307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.260122 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.260435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.260412 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.262373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.262355 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.262735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.262714 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.262783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.262754 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.262836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.262817 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.272769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.272750 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxm64\" (UniqueName: \"kubernetes.io/projected/dbe1dd46-1d2d-45f5-b549-7426d8bedbda-kube-api-access-gxm64\") pod \"opendatahub-operator-controller-manager-8865cc5f6-6mcwq\" (UID: \"dbe1dd46-1d2d-45f5-b549-7426d8bedbda\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.351069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.349066 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:48.650559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.650532 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq"] Nov 03 20:06:48.653328 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:06:48.653298 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe1dd46_1d2d_45f5_b549_7426d8bedbda.slice/crio-c1c4cd7196d5b673be2670795b30b7ab58754da6f276402e45b9d2facb766415 WatchSource:0}: Error finding container c1c4cd7196d5b673be2670795b30b7ab58754da6f276402e45b9d2facb766415: Status 404 returned error can't find the container with id c1c4cd7196d5b673be2670795b30b7ab58754da6f276402e45b9d2facb766415 Nov 03 20:06:48.654816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:48.654799 5815 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:06:49.364562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:49.364528 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" event={"ID":"dbe1dd46-1d2d-45f5-b549-7426d8bedbda","Type":"ContainerStarted","Data":"c1c4cd7196d5b673be2670795b30b7ab58754da6f276402e45b9d2facb766415"} Nov 03 20:06:52.383128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.383096 5815 generic.go:356] "Generic (PLEG): container finished" podID="dbe1dd46-1d2d-45f5-b549-7426d8bedbda" containerID="90ac494e9ad548ba7db21148b18e91ae72e630406013f0fda64284b61e32ecb4" exitCode=0 Nov 03 20:06:52.383531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.383139 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" event={"ID":"dbe1dd46-1d2d-45f5-b549-7426d8bedbda","Type":"ContainerDied","Data":"90ac494e9ad548ba7db21148b18e91ae72e630406013f0fda64284b61e32ecb4"} Nov 03 20:06:52.679753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.679722 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl"] Nov 03 20:06:52.683642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.683627 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.692768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.692749 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:06:52.692875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.692819 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:06:52.692875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.692867 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9brjz\"" Nov 03 20:06:52.699683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.699662 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl"] Nov 03 20:06:52.794403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.794379 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhhkr\" (UniqueName: \"kubernetes.io/projected/65351456-d78b-4b91-a300-9505231226fe-kube-api-access-bhhkr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl\" (UID: \"65351456-d78b-4b91-a300-9505231226fe\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.794571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.794443 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65351456-d78b-4b91-a300-9505231226fe-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl\" (UID: \"65351456-d78b-4b91-a300-9505231226fe\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.895501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.895470 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhhkr\" (UniqueName: \"kubernetes.io/projected/65351456-d78b-4b91-a300-9505231226fe-kube-api-access-bhhkr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl\" (UID: \"65351456-d78b-4b91-a300-9505231226fe\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.895617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.895562 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65351456-d78b-4b91-a300-9505231226fe-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl\" (UID: \"65351456-d78b-4b91-a300-9505231226fe\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.895884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.895858 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65351456-d78b-4b91-a300-9505231226fe-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl\" (UID: \"65351456-d78b-4b91-a300-9505231226fe\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.910062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.910035 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhhkr\" (UniqueName: \"kubernetes.io/projected/65351456-d78b-4b91-a300-9505231226fe-kube-api-access-bhhkr\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl\" (UID: \"65351456-d78b-4b91-a300-9505231226fe\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:52.996628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:52.996600 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" Nov 03 20:06:53.327663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:53.327588 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl"] Nov 03 20:06:53.331760 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:06:53.331735 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65351456_d78b_4b91_a300_9505231226fe.slice/crio-60c7f51435080a6cfc9dc7a59400e29eeb1783f236961cc2024d5de082a4f1db WatchSource:0}: Error finding container 60c7f51435080a6cfc9dc7a59400e29eeb1783f236961cc2024d5de082a4f1db: Status 404 returned error can't find the container with id 60c7f51435080a6cfc9dc7a59400e29eeb1783f236961cc2024d5de082a4f1db Nov 03 20:06:53.397295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:53.397248 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" event={"ID":"dbe1dd46-1d2d-45f5-b549-7426d8bedbda","Type":"ContainerStarted","Data":"6730eb5540aa99714d86f5e17a9c65f32a0ee290af1fc3f2b15ea9f4241dc5e8"} Nov 03 20:06:53.397821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:53.397424 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:06:53.398823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:53.398799 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" event={"ID":"65351456-d78b-4b91-a300-9505231226fe","Type":"ContainerStarted","Data":"60c7f51435080a6cfc9dc7a59400e29eeb1783f236961cc2024d5de082a4f1db"} Nov 03 20:06:53.450281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:53.450216 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" podStartSLOduration=1.568936841 podStartE2EDuration="5.450203348s" totalImagesPullingTime="3.454183649s" totalInitContainerRuntime="427.082858ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:53.447878709 +0000 UTC m=+322.978224380" watchObservedRunningTime="2026-11-03 20:06:53.450203348 +0000 UTC m=+322.980549019" Nov 03 20:06:56.417114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:56.417081 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" event={"ID":"65351456-d78b-4b91-a300-9505231226fe","Type":"ContainerStarted","Data":"25860ccba26d582d7f7f3d361f8a34b7138789122e0f361962516fafb310e88a"} Nov 03 20:06:56.487882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:06:56.487789 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-9ltxl" podStartSLOduration=1.588968326 podStartE2EDuration="4.487776405s" totalImagesPullingTime="2.898808079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:56.485077578 +0000 UTC m=+326.015423246" watchObservedRunningTime="2026-11-03 20:06:56.487776405 +0000 UTC m=+326.018122076" Nov 03 20:07:01.970214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:01.970179 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s"] Nov 03 20:07:01.974584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:01.974563 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:01.982568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:01.982497 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:07:01.982680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:01.982576 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:07:01.982727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:01.982677 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8td8b\"" Nov 03 20:07:01.984564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:01.984541 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s"] Nov 03 20:07:02.069423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:02.069396 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc2rg\" (UniqueName: \"kubernetes.io/projected/9da9ab34-02dc-458e-84d7-e9522d44c516-kube-api-access-vc2rg\") pod \"limitador-operator-controller-manager-5778bb4449-h946s\" (UID: \"9da9ab34-02dc-458e-84d7-e9522d44c516\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:02.170659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:02.170633 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc2rg\" (UniqueName: \"kubernetes.io/projected/9da9ab34-02dc-458e-84d7-e9522d44c516-kube-api-access-vc2rg\") pod \"limitador-operator-controller-manager-5778bb4449-h946s\" (UID: \"9da9ab34-02dc-458e-84d7-e9522d44c516\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:02.183174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:02.183147 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc2rg\" (UniqueName: \"kubernetes.io/projected/9da9ab34-02dc-458e-84d7-e9522d44c516-kube-api-access-vc2rg\") pod \"limitador-operator-controller-manager-5778bb4449-h946s\" (UID: \"9da9ab34-02dc-458e-84d7-e9522d44c516\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:02.286973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:02.286904 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:02.559931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:02.559907 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s"] Nov 03 20:07:02.562473 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:02.562440 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da9ab34_02dc_458e_84d7_e9522d44c516.slice/crio-dab06dcd9fccb93af1174d77f5db2769d10b167f36de2d51a07bd279fb59f52c WatchSource:0}: Error finding container dab06dcd9fccb93af1174d77f5db2769d10b167f36de2d51a07bd279fb59f52c: Status 404 returned error can't find the container with id dab06dcd9fccb93af1174d77f5db2769d10b167f36de2d51a07bd279fb59f52c Nov 03 20:07:03.450920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:03.450884 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" event={"ID":"9da9ab34-02dc-458e-84d7-e9522d44c516","Type":"ContainerStarted","Data":"dab06dcd9fccb93af1174d77f5db2769d10b167f36de2d51a07bd279fb59f52c"} Nov 03 20:07:04.407611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:04.407582 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-6mcwq" Nov 03 20:07:06.467940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.467900 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" event={"ID":"9da9ab34-02dc-458e-84d7-e9522d44c516","Type":"ContainerStarted","Data":"19c05330c1e9592c4789d4ac0766c6f6205cea2785e362fa3acf2150873142fd"} Nov 03 20:07:06.468314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.467980 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:06.491679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.491637 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" podStartSLOduration=2.622202339 podStartE2EDuration="5.491625459s" totalImagesPullingTime="2.86942312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:06.489936023 +0000 UTC m=+336.020281695" watchObservedRunningTime="2026-11-03 20:07:06.491625459 +0000 UTC m=+336.021971131" Nov 03 20:07:06.702534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.702480 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8"] Nov 03 20:07:06.706117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.706099 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:06.713402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.713384 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lbttq\"" Nov 03 20:07:06.722592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.722538 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8"] Nov 03 20:07:06.808044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.808024 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfxx\" (UniqueName: \"kubernetes.io/projected/2276a55f-e1a9-406a-994e-ace6113b9d42-kube-api-access-nrfxx\") pod \"authorino-operator-7b7bb4bc77-bx9l8\" (UID: \"2276a55f-e1a9-406a-994e-ace6113b9d42\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:06.909312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.909289 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfxx\" (UniqueName: \"kubernetes.io/projected/2276a55f-e1a9-406a-994e-ace6113b9d42-kube-api-access-nrfxx\") pod \"authorino-operator-7b7bb4bc77-bx9l8\" (UID: \"2276a55f-e1a9-406a-994e-ace6113b9d42\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:06.923852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:06.923824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfxx\" (UniqueName: \"kubernetes.io/projected/2276a55f-e1a9-406a-994e-ace6113b9d42-kube-api-access-nrfxx\") pod \"authorino-operator-7b7bb4bc77-bx9l8\" (UID: \"2276a55f-e1a9-406a-994e-ace6113b9d42\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:07.019116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:07.019065 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:07.295526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:07.295484 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8"] Nov 03 20:07:07.297717 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:07.297689 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2276a55f_e1a9_406a_994e_ace6113b9d42.slice/crio-a5843039200dec20ffc058746ca055705bcaac4c1ea356941f7dd2745fa6c9e6 WatchSource:0}: Error finding container a5843039200dec20ffc058746ca055705bcaac4c1ea356941f7dd2745fa6c9e6: Status 404 returned error can't find the container with id a5843039200dec20ffc058746ca055705bcaac4c1ea356941f7dd2745fa6c9e6 Nov 03 20:07:07.474311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:07.474277 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" event={"ID":"2276a55f-e1a9-406a-994e-ace6113b9d42","Type":"ContainerStarted","Data":"a5843039200dec20ffc058746ca055705bcaac4c1ea356941f7dd2745fa6c9e6"} Nov 03 20:07:09.485902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:09.485866 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" event={"ID":"2276a55f-e1a9-406a-994e-ace6113b9d42","Type":"ContainerStarted","Data":"1b00b16c27da2450f925edf768a5261ed1ebbc7dfdcdd0a1faa2f47eb8562982"} Nov 03 20:07:09.486266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:09.485957 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:09.511735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:09.511658 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" podStartSLOduration=1.5533972729999999 podStartE2EDuration="3.511645285s" totalImagesPullingTime="1.958248012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:09.510903605 +0000 UTC m=+339.041249277" watchObservedRunningTime="2026-11-03 20:07:09.511645285 +0000 UTC m=+339.041990956" Nov 03 20:07:10.458052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.458017 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4"] Nov 03 20:07:10.461570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.461554 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:10.467199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.467177 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9vt2r\"" Nov 03 20:07:10.469785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.469768 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 20:07:10.475717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.475693 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4"] Nov 03 20:07:10.642116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.642075 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhtq\" (UniqueName: \"kubernetes.io/projected/df4b5345-2b3d-425d-a9f9-2dc3f382fcf7-kube-api-access-vlhtq\") pod \"dns-operator-controller-manager-8557cbdb84-jgwq4\" (UID: \"df4b5345-2b3d-425d-a9f9-2dc3f382fcf7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:10.743174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.743138 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhtq\" (UniqueName: \"kubernetes.io/projected/df4b5345-2b3d-425d-a9f9-2dc3f382fcf7-kube-api-access-vlhtq\") pod \"dns-operator-controller-manager-8557cbdb84-jgwq4\" (UID: \"df4b5345-2b3d-425d-a9f9-2dc3f382fcf7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:10.758034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.758006 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhtq\" (UniqueName: \"kubernetes.io/projected/df4b5345-2b3d-425d-a9f9-2dc3f382fcf7-kube-api-access-vlhtq\") pod \"dns-operator-controller-manager-8557cbdb84-jgwq4\" (UID: \"df4b5345-2b3d-425d-a9f9-2dc3f382fcf7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:10.773960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:10.773938 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:11.060732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:11.060710 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4"] Nov 03 20:07:11.062624 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:11.062601 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4b5345_2b3d_425d_a9f9_2dc3f382fcf7.slice/crio-eb9078a3287d467da73a505bb96857cddef2d3230469c0089345f220d0e844a8 WatchSource:0}: Error finding container eb9078a3287d467da73a505bb96857cddef2d3230469c0089345f220d0e844a8: Status 404 returned error can't find the container with id eb9078a3287d467da73a505bb96857cddef2d3230469c0089345f220d0e844a8 Nov 03 20:07:11.498157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:11.498128 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" event={"ID":"df4b5345-2b3d-425d-a9f9-2dc3f382fcf7","Type":"ContainerStarted","Data":"eb9078a3287d467da73a505bb96857cddef2d3230469c0089345f220d0e844a8"} Nov 03 20:07:13.511569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:13.511537 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" event={"ID":"df4b5345-2b3d-425d-a9f9-2dc3f382fcf7","Type":"ContainerStarted","Data":"14538346dc595c17afc13fdac9e00b8c115095ca792bec25936aec0a6ec6ea5d"} Nov 03 20:07:13.512011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:13.511598 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:13.532199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:13.532154 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" podStartSLOduration=1.568345898 podStartE2EDuration="3.532140706s" totalImagesPullingTime="1.963794808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:13.531472244 +0000 UTC m=+343.061817917" watchObservedRunningTime="2026-11-03 20:07:13.532140706 +0000 UTC m=+343.062486380" Nov 03 20:07:13.849167 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:07:13.849082 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:07:15.781712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.781684 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh"] Nov 03 20:07:15.785319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.785298 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:15.813485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.813446 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh"] Nov 03 20:07:15.818389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.818367 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pl6rl\"" Nov 03 20:07:15.882391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.882361 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k4psh\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:15.882572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.882396 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmdgl\" (UniqueName: \"kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl\") pod \"kuadrant-operator-controller-manager-678d87766f-k4psh\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:15.983055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.983023 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k4psh\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:15.983201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.983069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmdgl\" (UniqueName: \"kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl\") pod \"kuadrant-operator-controller-manager-678d87766f-k4psh\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:15.983435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.983415 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-k4psh\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:15.996897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:15.996871 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmdgl\" (UniqueName: \"kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl\") pod \"kuadrant-operator-controller-manager-678d87766f-k4psh\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:16.097414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:16.097359 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:16.436136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:16.436074 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh"] Nov 03 20:07:16.438352 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:16.438328 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79d3bb8_3533_4def_a7d4_2fba2b589a0f.slice/crio-91ec1d1fef25cafe132f1d2bd47c7120b2d4a0130ccbb43dfc6565abfcfe79fd WatchSource:0}: Error finding container 91ec1d1fef25cafe132f1d2bd47c7120b2d4a0130ccbb43dfc6565abfcfe79fd: Status 404 returned error can't find the container with id 91ec1d1fef25cafe132f1d2bd47c7120b2d4a0130ccbb43dfc6565abfcfe79fd Nov 03 20:07:16.528703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:16.528673 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" event={"ID":"e79d3bb8-3533-4def-a7d4-2fba2b589a0f","Type":"ContainerStarted","Data":"91ec1d1fef25cafe132f1d2bd47c7120b2d4a0130ccbb43dfc6565abfcfe79fd"} Nov 03 20:07:17.477035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:17.477010 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h946s" Nov 03 20:07:20.494274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:20.494249 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-bx9l8" Nov 03 20:07:23.023320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.023279 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr"] Nov 03 20:07:23.026202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.026178 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.034877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.034843 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-q6lk7\"" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039154 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039193 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039254 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039284 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039320 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039361 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039421 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039446 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjfw\" (UniqueName: \"kubernetes.io/projected/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-kube-api-access-drjfw\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.039691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.039474 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.043664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.043639 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr"] Nov 03 20:07:23.140462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140431 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140466 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjfw\" (UniqueName: \"kubernetes.io/projected/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-kube-api-access-drjfw\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140490 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140544 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140563 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140606 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140637 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140675 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.140720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.140718 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.141061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.141032 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.141123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.141078 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.141168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.141149 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.141359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.141333 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.141478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.141383 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.143722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.143701 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.143911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.143894 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.153044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.153019 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.153144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.153129 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjfw\" (UniqueName: \"kubernetes.io/projected/e5cb68d0-ef10-4bc6-9677-5eff1054ca0f-kube-api-access-drjfw\") pod \"data-science-gateway-data-science-gateway-class-76f56c5945zkwwr\" (UID: \"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.342307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.342258 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:23.569986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.569953 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" event={"ID":"e79d3bb8-3533-4def-a7d4-2fba2b589a0f","Type":"ContainerStarted","Data":"831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8"} Nov 03 20:07:23.570190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.570062 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:23.594828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.594734 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" podStartSLOduration=2.096023267 podStartE2EDuration="8.594715376s" totalImagesPullingTime="6.498692109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:23.592088083 +0000 UTC m=+353.122433758" watchObservedRunningTime="2026-11-03 20:07:23.594715376 +0000 UTC m=+353.125061049" Nov 03 20:07:23.656432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.656411 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr"] Nov 03 20:07:23.658530 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:23.658492 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cb68d0_ef10_4bc6_9677_5eff1054ca0f.slice/crio-b077071ad9548e09e4e30d58e508de361e16e92a2030847bdc2055387b7cb25b WatchSource:0}: Error finding container b077071ad9548e09e4e30d58e508de361e16e92a2030847bdc2055387b7cb25b: Status 404 returned error can't find the container with id b077071ad9548e09e4e30d58e508de361e16e92a2030847bdc2055387b7cb25b Nov 03 20:07:23.661322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.661292 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:07:23.661395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.661351 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:07:23.661395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:23.661382 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:07:24.520097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.520066 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-jgwq4" Nov 03 20:07:24.589025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.588986 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" event={"ID":"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f","Type":"ContainerStarted","Data":"2816233b4c784bc4fbfe0c10a8b42cc37c509b0f9f5a8fc053450f71ed9b93bb"} Nov 03 20:07:24.589218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.589048 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" event={"ID":"e5cb68d0-ef10-4bc6-9677-5eff1054ca0f","Type":"ContainerStarted","Data":"b077071ad9548e09e4e30d58e508de361e16e92a2030847bdc2055387b7cb25b"} Nov 03 20:07:24.611894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.611828 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" podStartSLOduration=2.6118114759999997 podStartE2EDuration="2.611811476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:24.61103335 +0000 UTC m=+354.141379022" watchObservedRunningTime="2026-11-03 20:07:24.611811476 +0000 UTC m=+354.142157149" Nov 03 20:07:24.855863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.855790 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:07:24.858820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.858798 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:24.868723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.868702 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nh6tb\"" Nov 03 20:07:24.868859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.868722 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:07:24.875496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.875468 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:07:24.894755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.894724 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:07:24.955652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.955595 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpvr\" (UniqueName: \"kubernetes.io/projected/683dd503-5b16-4acc-9d0f-36fd1baf048e-kube-api-access-ffpvr\") pod \"limitador-limitador-7c9b87d574-q9gdw\" (UID: \"683dd503-5b16-4acc-9d0f-36fd1baf048e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:24.955807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:24.955784 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/683dd503-5b16-4acc-9d0f-36fd1baf048e-config-file\") pod \"limitador-limitador-7c9b87d574-q9gdw\" (UID: \"683dd503-5b16-4acc-9d0f-36fd1baf048e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:25.056888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.056856 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/683dd503-5b16-4acc-9d0f-36fd1baf048e-config-file\") pod \"limitador-limitador-7c9b87d574-q9gdw\" (UID: \"683dd503-5b16-4acc-9d0f-36fd1baf048e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:25.057064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.056938 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpvr\" (UniqueName: \"kubernetes.io/projected/683dd503-5b16-4acc-9d0f-36fd1baf048e-kube-api-access-ffpvr\") pod \"limitador-limitador-7c9b87d574-q9gdw\" (UID: \"683dd503-5b16-4acc-9d0f-36fd1baf048e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:25.057599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.057577 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/683dd503-5b16-4acc-9d0f-36fd1baf048e-config-file\") pod \"limitador-limitador-7c9b87d574-q9gdw\" (UID: \"683dd503-5b16-4acc-9d0f-36fd1baf048e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:25.075844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.075813 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpvr\" (UniqueName: \"kubernetes.io/projected/683dd503-5b16-4acc-9d0f-36fd1baf048e-kube-api-access-ffpvr\") pod \"limitador-limitador-7c9b87d574-q9gdw\" (UID: \"683dd503-5b16-4acc-9d0f-36fd1baf048e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:25.131488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.131412 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h5gbv"] Nov 03 20:07:25.134022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.134005 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:25.143454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.143431 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h5gbv"] Nov 03 20:07:25.157489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.157453 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8df\" (UniqueName: \"kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df\") pod \"authorino-57f7f8659f-h5gbv\" (UID: \"d7adb590-71fd-4cc6-9b2d-3440ef461158\") " pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:25.171396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.171370 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:25.223538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.223329 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-44qgz\"" Nov 03 20:07:25.258206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.258180 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8df\" (UniqueName: \"kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df\") pod \"authorino-57f7f8659f-h5gbv\" (UID: \"d7adb590-71fd-4cc6-9b2d-3440ef461158\") " pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:25.277424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.277397 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8df\" (UniqueName: \"kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df\") pod \"authorino-57f7f8659f-h5gbv\" (UID: \"d7adb590-71fd-4cc6-9b2d-3440ef461158\") " pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:25.342724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.342693 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:25.347628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.347605 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:25.450278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.450255 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:07:25.452648 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:25.452620 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683dd503_5b16_4acc_9d0f_36fd1baf048e.slice/crio-4bac1936cffdd04d41c62aec875b3cec3435b348d939145ef2622148b89c94a8 WatchSource:0}: Error finding container 4bac1936cffdd04d41c62aec875b3cec3435b348d939145ef2622148b89c94a8: Status 404 returned error can't find the container with id 4bac1936cffdd04d41c62aec875b3cec3435b348d939145ef2622148b89c94a8 Nov 03 20:07:25.461962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.461941 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:25.596142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.596107 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" event={"ID":"683dd503-5b16-4acc-9d0f-36fd1baf048e","Type":"ContainerStarted","Data":"4bac1936cffdd04d41c62aec875b3cec3435b348d939145ef2622148b89c94a8"} Nov 03 20:07:25.596627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.596462 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:25.597268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.597247 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-76f56c5945zkwwr" Nov 03 20:07:25.734492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:25.734263 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h5gbv"] Nov 03 20:07:25.736828 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:25.736799 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7adb590_71fd_4cc6_9b2d_3440ef461158.slice/crio-82f201f1a78d12eb6462423b5e17975136e38d8795d470e71f972f332045ccd7 WatchSource:0}: Error finding container 82f201f1a78d12eb6462423b5e17975136e38d8795d470e71f972f332045ccd7: Status 404 returned error can't find the container with id 82f201f1a78d12eb6462423b5e17975136e38d8795d470e71f972f332045ccd7 Nov 03 20:07:26.602500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:26.602470 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" event={"ID":"d7adb590-71fd-4cc6-9b2d-3440ef461158","Type":"ContainerStarted","Data":"82f201f1a78d12eb6462423b5e17975136e38d8795d470e71f972f332045ccd7"} Nov 03 20:07:27.610832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:27.610792 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" event={"ID":"683dd503-5b16-4acc-9d0f-36fd1baf048e","Type":"ContainerStarted","Data":"57fdd287a0299ce7ac04dea549a31d09c8cca1a042edd111a88d8502696fac4f"} Nov 03 20:07:27.611269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:27.610862 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:27.633934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:27.631900 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" podStartSLOduration=2.450427665 podStartE2EDuration="3.631884114s" totalImagesPullingTime="1.181456449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:27.62999538 +0000 UTC m=+357.160341054" watchObservedRunningTime="2026-11-03 20:07:27.631884114 +0000 UTC m=+357.162229789" Nov 03 20:07:28.617103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:28.617069 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" event={"ID":"d7adb590-71fd-4cc6-9b2d-3440ef461158","Type":"ContainerStarted","Data":"fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63"} Nov 03 20:07:28.636868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:28.636814 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" podStartSLOduration=1.3351133499999999 podStartE2EDuration="3.636800346s" totalImagesPullingTime="2.301686996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:28.635401755 +0000 UTC m=+358.165747427" watchObservedRunningTime="2026-11-03 20:07:28.636800346 +0000 UTC m=+358.167146021" Nov 03 20:07:30.393154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:30.393121 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h5gbv"] Nov 03 20:07:30.627375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:30.627331 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" podUID="d7adb590-71fd-4cc6-9b2d-3440ef461158" containerName="authorino" containerID="cri-o://fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63" gracePeriod=30 Nov 03 20:07:30.886423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:30.886401 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:30.910875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:30.910808 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df\" (UniqueName: \"kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df\") pod \"d7adb590-71fd-4cc6-9b2d-3440ef461158\" (UID: \"d7adb590-71fd-4cc6-9b2d-3440ef461158\") " Nov 03 20:07:30.913056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:30.913032 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df" pod "d7adb590-71fd-4cc6-9b2d-3440ef461158" (UID: "d7adb590-71fd-4cc6-9b2d-3440ef461158"). InnerVolumeSpecName "kube-api-access-5x8df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:31.012201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.012167 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x8df\" (UniqueName: \"kubernetes.io/projected/d7adb590-71fd-4cc6-9b2d-3440ef461158-kube-api-access-5x8df\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:31.633399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.633363 5815 generic.go:356] "Generic (PLEG): container finished" podID="d7adb590-71fd-4cc6-9b2d-3440ef461158" containerID="fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63" exitCode=0 Nov 03 20:07:31.633821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.633409 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" Nov 03 20:07:31.633821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.633453 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" event={"ID":"d7adb590-71fd-4cc6-9b2d-3440ef461158","Type":"ContainerDied","Data":"fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63"} Nov 03 20:07:31.633821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.633518 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-h5gbv" event={"ID":"d7adb590-71fd-4cc6-9b2d-3440ef461158","Type":"ContainerDied","Data":"82f201f1a78d12eb6462423b5e17975136e38d8795d470e71f972f332045ccd7"} Nov 03 20:07:31.633821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.633544 5815 scope.go:122] "RemoveContainer" containerID="fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63" Nov 03 20:07:31.642923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.642907 5815 scope.go:122] "RemoveContainer" containerID="fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63" Nov 03 20:07:31.643220 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:07:31.643202 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63\": container with ID starting with fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63 not found: ID does not exist" containerID="fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63" Nov 03 20:07:31.643270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.643231 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63"} err="failed to get container status \"fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63\": rpc error: code = NotFound desc = could not find container \"fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63\": container with ID starting with fab4f51aecbed7c791c0e0adc2290ffbeb8eb2e207bfe6c1e4d77c870a4bda63 not found: ID does not exist" Nov 03 20:07:31.655598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.655575 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h5gbv"] Nov 03 20:07:31.661660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:31.661641 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-h5gbv"] Nov 03 20:07:32.587418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:32.587377 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7adb590-71fd-4cc6-9b2d-3440ef461158" path="/var/lib/kubelet/pods/d7adb590-71fd-4cc6-9b2d-3440ef461158/volumes" Nov 03 20:07:34.591249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:34.591225 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 20:07:38.618157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:38.618082 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-q9gdw" Nov 03 20:07:43.851243 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:07:43.851204 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:07:45.625090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.625057 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn"] Nov 03 20:07:45.625641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.625469 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d7adb590-71fd-4cc6-9b2d-3440ef461158" containerName="authorino" Nov 03 20:07:45.625641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.625494 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7adb590-71fd-4cc6-9b2d-3440ef461158" containerName="authorino" Nov 03 20:07:45.625641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.625629 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7adb590-71fd-4cc6-9b2d-3440ef461158" containerName="authorino" Nov 03 20:07:45.630115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.630097 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.634455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.634428 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:07:45.634600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.634528 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:07:45.635772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.635754 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-2s4t6\"" Nov 03 20:07:45.635909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.635818 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 20:07:45.640777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.640756 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn"] Nov 03 20:07:45.727423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.727387 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9lt9\" (UniqueName: \"kubernetes.io/projected/e5598e3b-f89f-47d0-a9bb-6099846c5ee2-kube-api-access-w9lt9\") pod \"postgres-7cdbf6dc44-bs6vn\" (UID: \"e5598e3b-f89f-47d0-a9bb-6099846c5ee2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.727616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.727530 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5598e3b-f89f-47d0-a9bb-6099846c5ee2-data\") pod \"postgres-7cdbf6dc44-bs6vn\" (UID: \"e5598e3b-f89f-47d0-a9bb-6099846c5ee2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.828339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.828298 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5598e3b-f89f-47d0-a9bb-6099846c5ee2-data\") pod \"postgres-7cdbf6dc44-bs6vn\" (UID: \"e5598e3b-f89f-47d0-a9bb-6099846c5ee2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.828556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.828377 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9lt9\" (UniqueName: \"kubernetes.io/projected/e5598e3b-f89f-47d0-a9bb-6099846c5ee2-kube-api-access-w9lt9\") pod \"postgres-7cdbf6dc44-bs6vn\" (UID: \"e5598e3b-f89f-47d0-a9bb-6099846c5ee2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.828822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.828796 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5598e3b-f89f-47d0-a9bb-6099846c5ee2-data\") pod \"postgres-7cdbf6dc44-bs6vn\" (UID: \"e5598e3b-f89f-47d0-a9bb-6099846c5ee2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.885866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.885788 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9lt9\" (UniqueName: \"kubernetes.io/projected/e5598e3b-f89f-47d0-a9bb-6099846c5ee2-kube-api-access-w9lt9\") pod \"postgres-7cdbf6dc44-bs6vn\" (UID: \"e5598e3b-f89f-47d0-a9bb-6099846c5ee2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:45.955746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:45.955722 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:46.235413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:46.235386 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn"] Nov 03 20:07:46.237727 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:07:46.237700 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5598e3b_f89f_47d0_a9bb_6099846c5ee2.slice/crio-7a042a3de597bc71a0077b412568eac992240bc0d5783833e49f490d1d9d3429 WatchSource:0}: Error finding container 7a042a3de597bc71a0077b412568eac992240bc0d5783833e49f490d1d9d3429: Status 404 returned error can't find the container with id 7a042a3de597bc71a0077b412568eac992240bc0d5783833e49f490d1d9d3429 Nov 03 20:07:46.710927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:46.710896 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" event={"ID":"e5598e3b-f89f-47d0-a9bb-6099846c5ee2","Type":"ContainerStarted","Data":"7a042a3de597bc71a0077b412568eac992240bc0d5783833e49f490d1d9d3429"} Nov 03 20:07:52.750106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:52.750066 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" event={"ID":"e5598e3b-f89f-47d0-a9bb-6099846c5ee2","Type":"ContainerStarted","Data":"14286dd3092b8eed980827cef39f0e398d13de99e36c7918fbc8e2ab7af3be33"} Nov 03 20:07:52.750557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:52.750208 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:07:52.768843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:52.768801 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" podStartSLOduration=2.268426881 podStartE2EDuration="7.768788434s" totalImagesPullingTime="5.500361553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:52.767693712 +0000 UTC m=+382.298039398" watchObservedRunningTime="2026-11-03 20:07:52.768788434 +0000 UTC m=+382.299134106" Nov 03 20:07:58.785782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:07:58.785748 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-bs6vn" Nov 03 20:08:13.849827 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:08:13.849783 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:08:18.873192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.873162 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp"] Nov 03 20:08:18.876964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.876944 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:18.885081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.884602 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:08:18.887135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.887117 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:08:18.890026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.890000 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp"] Nov 03 20:08:18.904044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.904004 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 20:08:18.904172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.904037 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h9jc6\"" Nov 03 20:08:18.996635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.996555 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hmkn\" (UniqueName: \"kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn\") pod \"llmisvc-controller-manager-869d5dcc5d-sz7qp\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:18.996635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:18.996610 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sz7qp\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:19.098038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.097955 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hmkn\" (UniqueName: \"kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn\") pod \"llmisvc-controller-manager-869d5dcc5d-sz7qp\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:19.098349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.098321 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sz7qp\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:19.101338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.101318 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-sz7qp\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:19.111782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.111763 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hmkn\" (UniqueName: \"kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn\") pod \"llmisvc-controller-manager-869d5dcc5d-sz7qp\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:19.191462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.191439 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:19.480357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.480333 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp"] Nov 03 20:08:19.482565 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:08:19.482531 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7adad751_6492_456c_869b_731eac0154a9.slice/crio-ef6e5755fd290b13876a2b2e497ec0e9badd324d685805446c74b90bc10c2d62 WatchSource:0}: Error finding container ef6e5755fd290b13876a2b2e497ec0e9badd324d685805446c74b90bc10c2d62: Status 404 returned error can't find the container with id ef6e5755fd290b13876a2b2e497ec0e9badd324d685805446c74b90bc10c2d62 Nov 03 20:08:19.891097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:19.891067 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" event={"ID":"7adad751-6492-456c-869b-731eac0154a9","Type":"ContainerStarted","Data":"ef6e5755fd290b13876a2b2e497ec0e9badd324d685805446c74b90bc10c2d62"} Nov 03 20:08:23.914751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:23.914718 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" event={"ID":"7adad751-6492-456c-869b-731eac0154a9","Type":"ContainerStarted","Data":"8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215"} Nov 03 20:08:23.915141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:23.914781 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:23.933881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:23.933836 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" podStartSLOduration=2.437851111 podStartE2EDuration="5.933823368s" totalImagesPullingTime="3.495972257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:23.932757403 +0000 UTC m=+413.463103074" watchObservedRunningTime="2026-11-03 20:08:23.933823368 +0000 UTC m=+413.464169041" Nov 03 20:08:43.851153 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:08:43.851117 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:08:44.690250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:44.690219 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:08:54.922145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:54.922112 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:08:55.081210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.081176 5815 generic.go:356] "Generic (PLEG): container finished" podID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerID="b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347" exitCode=0 Nov 03 20:08:55.081361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.081232 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" event={"ID":"369568fe-f4a1-4dd5-904a-eb30dae71013","Type":"ContainerDied","Data":"b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347"} Nov 03 20:08:55.081658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.081638 5815 scope.go:122] "RemoveContainer" containerID="b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347" Nov 03 20:08:55.082768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.082740 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:08:55.082837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.082808 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:08:55.082881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.082842 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:08:55.379421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.379390 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w94ts"] Nov 03 20:08:55.383302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.383280 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:55.390387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.390365 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w94ts"] Nov 03 20:08:55.390679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.390661 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-44qgz\"" Nov 03 20:08:55.470700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.470664 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w94ts"] Nov 03 20:08:55.471061 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:08:55.471033 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zjjq8], unattached volumes=[], failed to process volumes=[kube-api-access-zjjq8]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-w94ts" podUID="b898ff28-8804-4bce-a7ba-916bb5e96a45" Nov 03 20:08:55.501758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.499562 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjq8\" (UniqueName: \"kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8\") pod \"authorino-6f6944bcdd-w94ts\" (UID: \"b898ff28-8804-4bce-a7ba-916bb5e96a45\") " pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:55.501758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.499600 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2jvfj"] Nov 03 20:08:55.503755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.503729 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.508005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.507983 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:08:55.509212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.509190 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2jvfj"] Nov 03 20:08:55.600217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.600181 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dct\" (UniqueName: \"kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct\") pod \"authorino-6889cbd8c4-2jvfj\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.600384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.600320 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjq8\" (UniqueName: \"kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8\") pod \"authorino-6f6944bcdd-w94ts\" (UID: \"b898ff28-8804-4bce-a7ba-916bb5e96a45\") " pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:55.600384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.600352 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert\") pod \"authorino-6889cbd8c4-2jvfj\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.613158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.613132 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjq8\" (UniqueName: \"kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8\") pod \"authorino-6f6944bcdd-w94ts\" (UID: \"b898ff28-8804-4bce-a7ba-916bb5e96a45\") " pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:55.701186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.701092 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert\") pod \"authorino-6889cbd8c4-2jvfj\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.701186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.701141 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dct\" (UniqueName: \"kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct\") pod \"authorino-6889cbd8c4-2jvfj\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.704005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.703972 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert\") pod \"authorino-6889cbd8c4-2jvfj\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.713789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.713765 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dct\" (UniqueName: \"kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct\") pod \"authorino-6889cbd8c4-2jvfj\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:55.742336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.742311 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2jvfj"] Nov 03 20:08:55.742554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:55.742540 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:56.050562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.050538 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2jvfj"] Nov 03 20:08:56.052150 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:08:56.052118 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6184e2a_174a_4745_88a2_dfba9b8b0d03.slice/crio-95ef892efeef8303b88d6a96f3c67522b2a07137615f204b7415c5a824a321f1 WatchSource:0}: Error finding container 95ef892efeef8303b88d6a96f3c67522b2a07137615f204b7415c5a824a321f1: Status 404 returned error can't find the container with id 95ef892efeef8303b88d6a96f3c67522b2a07137615f204b7415c5a824a321f1 Nov 03 20:08:56.096543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.096488 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" event={"ID":"369568fe-f4a1-4dd5-904a-eb30dae71013","Type":"ContainerStarted","Data":"71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b"} Nov 03 20:08:56.097857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.097827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" event={"ID":"f6184e2a-174a-4745-88a2-dfba9b8b0d03","Type":"ContainerStarted","Data":"95ef892efeef8303b88d6a96f3c67522b2a07137615f204b7415c5a824a321f1"} Nov 03 20:08:56.097857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.097842 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:56.106195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.106175 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:56.204898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.204869 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8\" (UniqueName: \"kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8\") pod \"b898ff28-8804-4bce-a7ba-916bb5e96a45\" (UID: \"b898ff28-8804-4bce-a7ba-916bb5e96a45\") " Nov 03 20:08:56.207208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.207183 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8" pod "b898ff28-8804-4bce-a7ba-916bb5e96a45" (UID: "b898ff28-8804-4bce-a7ba-916bb5e96a45"). InnerVolumeSpecName "kube-api-access-zjjq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:56.210025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.210005 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:08:56.211195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.211170 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:08:56.211282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.211224 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 20:08:56.306449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.306367 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjq8\" (UniqueName: \"kubernetes.io/projected/b898ff28-8804-4bce-a7ba-916bb5e96a45-kube-api-access-zjjq8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:56.885631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:56.885604 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:08:57.106215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.106176 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" event={"ID":"f6184e2a-174a-4745-88a2-dfba9b8b0d03","Type":"ContainerStarted","Data":"5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af"} Nov 03 20:08:57.106731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.106221 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" podUID="f6184e2a-174a-4745-88a2-dfba9b8b0d03" containerName="authorino" containerID="cri-o://5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af" gracePeriod=30 Nov 03 20:08:57.106731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.106594 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w94ts" Nov 03 20:08:57.124687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.124646 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" podStartSLOduration=1.36162918 podStartE2EDuration="2.124633034s" totalImagesPullingTime="763.003854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:57.123560277 +0000 UTC m=+446.653905948" watchObservedRunningTime="2026-11-03 20:08:57.124633034 +0000 UTC m=+446.654978706" Nov 03 20:08:57.149294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.149269 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w94ts"] Nov 03 20:08:57.157980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.157950 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w94ts"] Nov 03 20:08:57.210825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.210754 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:08:57.210825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.210820 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 20:08:57.396063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.395995 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:57.415889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.415431 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert\" (UniqueName: \"kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert\") pod \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " Nov 03 20:08:57.415889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.415574 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct\" (UniqueName: \"kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct\") pod \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\" (UID: \"f6184e2a-174a-4745-88a2-dfba9b8b0d03\") " Nov 03 20:08:57.419180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.419148 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct" pod "f6184e2a-174a-4745-88a2-dfba9b8b0d03" (UID: "f6184e2a-174a-4745-88a2-dfba9b8b0d03"). InnerVolumeSpecName "kube-api-access-k6dct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:57.428363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.428333 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert" pod "f6184e2a-174a-4745-88a2-dfba9b8b0d03" (UID: "f6184e2a-174a-4745-88a2-dfba9b8b0d03"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:57.516397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.516357 5815 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f6184e2a-174a-4745-88a2-dfba9b8b0d03-tls-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:57.516397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:57.516383 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6dct\" (UniqueName: \"kubernetes.io/projected/f6184e2a-174a-4745-88a2-dfba9b8b0d03-kube-api-access-k6dct\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:58.113954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.113921 5815 generic.go:356] "Generic (PLEG): container finished" podID="f6184e2a-174a-4745-88a2-dfba9b8b0d03" containerID="5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af" exitCode=0 Nov 03 20:08:58.114426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.113974 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" Nov 03 20:08:58.114426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.114004 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" event={"ID":"f6184e2a-174a-4745-88a2-dfba9b8b0d03","Type":"ContainerDied","Data":"5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af"} Nov 03 20:08:58.114426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.114043 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-2jvfj" event={"ID":"f6184e2a-174a-4745-88a2-dfba9b8b0d03","Type":"ContainerDied","Data":"95ef892efeef8303b88d6a96f3c67522b2a07137615f204b7415c5a824a321f1"} Nov 03 20:08:58.114426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.114060 5815 scope.go:122] "RemoveContainer" containerID="5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af" Nov 03 20:08:58.123928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.123913 5815 scope.go:122] "RemoveContainer" containerID="5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af" Nov 03 20:08:58.124192 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:08:58.124173 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af\": container with ID starting with 5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af not found: ID does not exist" containerID="5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af" Nov 03 20:08:58.124235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.124201 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af"} err="failed to get container status \"5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af\": rpc error: code = NotFound desc = could not find container \"5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af\": container with ID starting with 5caef52c45364e8d3ef3d5604c9ac6394f89439ee7b445890a2ae68a3c36f5af not found: ID does not exist" Nov 03 20:08:58.150575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.150556 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2jvfj"] Nov 03 20:08:58.160276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.160255 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2jvfj"] Nov 03 20:08:58.210856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.210827 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:08:58.210996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.210874 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 20:08:58.593109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.593082 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b898ff28-8804-4bce-a7ba-916bb5e96a45" path="/var/lib/kubelet/pods/b898ff28-8804-4bce-a7ba-916bb5e96a45/volumes" Nov 03 20:08:58.593329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:58.593317 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6184e2a-174a-4745-88a2-dfba9b8b0d03" path="/var/lib/kubelet/pods/f6184e2a-174a-4745-88a2-dfba9b8b0d03/volumes" Nov 03 20:08:59.210233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:59.210201 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:08:59.210648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:08:59.210291 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 20:09:00.210908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:00.210873 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 03 20:09:00.211328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:00.210938 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 03 20:09:01.215122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:01.215090 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:09:01.216199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:01.216177 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:09:13.849623 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:09:13.849548 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:09:18.925905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.925875 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x"] Nov 03 20:09:18.926261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.926249 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f6184e2a-174a-4745-88a2-dfba9b8b0d03" containerName="authorino" Nov 03 20:09:18.926312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.926262 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6184e2a-174a-4745-88a2-dfba9b8b0d03" containerName="authorino" Nov 03 20:09:18.926347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.926341 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6184e2a-174a-4745-88a2-dfba9b8b0d03" containerName="authorino" Nov 03 20:09:18.929521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.929473 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:18.942000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.941981 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x"] Nov 03 20:09:18.988355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.988329 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09dae28a-2531-444a-bf0c-bbe040b3b92d-cert\") pod \"llmisvc-controller-manager-6466c554d8-q5w7x\" (UID: \"09dae28a-2531-444a-bf0c-bbe040b3b92d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:18.988482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:18.988390 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzdc\" (UniqueName: \"kubernetes.io/projected/09dae28a-2531-444a-bf0c-bbe040b3b92d-kube-api-access-wkzdc\") pod \"llmisvc-controller-manager-6466c554d8-q5w7x\" (UID: \"09dae28a-2531-444a-bf0c-bbe040b3b92d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:19.089149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:19.089120 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09dae28a-2531-444a-bf0c-bbe040b3b92d-cert\") pod \"llmisvc-controller-manager-6466c554d8-q5w7x\" (UID: \"09dae28a-2531-444a-bf0c-bbe040b3b92d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:19.089296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:19.089179 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzdc\" (UniqueName: \"kubernetes.io/projected/09dae28a-2531-444a-bf0c-bbe040b3b92d-kube-api-access-wkzdc\") pod \"llmisvc-controller-manager-6466c554d8-q5w7x\" (UID: \"09dae28a-2531-444a-bf0c-bbe040b3b92d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:19.091755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:19.091727 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09dae28a-2531-444a-bf0c-bbe040b3b92d-cert\") pod \"llmisvc-controller-manager-6466c554d8-q5w7x\" (UID: \"09dae28a-2531-444a-bf0c-bbe040b3b92d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:19.102476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:19.102453 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzdc\" (UniqueName: \"kubernetes.io/projected/09dae28a-2531-444a-bf0c-bbe040b3b92d-kube-api-access-wkzdc\") pod \"llmisvc-controller-manager-6466c554d8-q5w7x\" (UID: \"09dae28a-2531-444a-bf0c-bbe040b3b92d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:19.242906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:19.242836 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:19.517225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:19.517203 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x"] Nov 03 20:09:19.519481 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:09:19.519453 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dae28a_2531_444a_bf0c_bbe040b3b92d.slice/crio-604bde7e9ae019c89e6c199885ddb581cecaaf631136be918e584847add5e9cb WatchSource:0}: Error finding container 604bde7e9ae019c89e6c199885ddb581cecaaf631136be918e584847add5e9cb: Status 404 returned error can't find the container with id 604bde7e9ae019c89e6c199885ddb581cecaaf631136be918e584847add5e9cb Nov 03 20:09:20.233324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:20.233290 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" event={"ID":"09dae28a-2531-444a-bf0c-bbe040b3b92d","Type":"ContainerStarted","Data":"070367054eb0e484b1d119e28b0e031377c40daadc8e898dd17299cf251760d8"} Nov 03 20:09:20.233704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:20.233378 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" event={"ID":"09dae28a-2531-444a-bf0c-bbe040b3b92d","Type":"ContainerStarted","Data":"604bde7e9ae019c89e6c199885ddb581cecaaf631136be918e584847add5e9cb"} Nov 03 20:09:20.233704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:20.233400 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:20.254477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:20.254432 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" podStartSLOduration=1.890878512 podStartE2EDuration="2.254419575s" totalImagesPullingTime="363.541063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:20.252942095 +0000 UTC m=+469.783287768" watchObservedRunningTime="2026-11-03 20:09:20.254419575 +0000 UTC m=+469.784765310" Nov 03 20:09:43.851167 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:09:43.851128 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:09:51.240025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.239997 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-q5w7x" Nov 03 20:09:51.292472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.292445 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp"] Nov 03 20:09:51.292731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.292708 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" podUID="7adad751-6492-456c-869b-731eac0154a9" containerName="manager" containerID="cri-o://8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215" gracePeriod=30 Nov 03 20:09:51.544177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.544152 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:09:51.577271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.577237 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert\" (UniqueName: \"kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert\") pod \"7adad751-6492-456c-869b-731eac0154a9\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " Nov 03 20:09:51.577400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.577291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn\" (UniqueName: \"kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn\") pod \"7adad751-6492-456c-869b-731eac0154a9\" (UID: \"7adad751-6492-456c-869b-731eac0154a9\") " Nov 03 20:09:51.579651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.579616 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert" pod "7adad751-6492-456c-869b-731eac0154a9" (UID: "7adad751-6492-456c-869b-731eac0154a9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:51.579651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.579627 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn" pod "7adad751-6492-456c-869b-731eac0154a9" (UID: "7adad751-6492-456c-869b-731eac0154a9"). InnerVolumeSpecName "kube-api-access-2hmkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:51.678201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.678174 5815 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7adad751-6492-456c-869b-731eac0154a9-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:51.678201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:51.678199 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hmkn\" (UniqueName: \"kubernetes.io/projected/7adad751-6492-456c-869b-731eac0154a9-kube-api-access-2hmkn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:52.411602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.411567 5815 generic.go:356] "Generic (PLEG): container finished" podID="7adad751-6492-456c-869b-731eac0154a9" containerID="8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215" exitCode=0 Nov 03 20:09:52.412035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.411615 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" Nov 03 20:09:52.412035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.411648 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" event={"ID":"7adad751-6492-456c-869b-731eac0154a9","Type":"ContainerDied","Data":"8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215"} Nov 03 20:09:52.412035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.411697 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp" event={"ID":"7adad751-6492-456c-869b-731eac0154a9","Type":"ContainerDied","Data":"ef6e5755fd290b13876a2b2e497ec0e9badd324d685805446c74b90bc10c2d62"} Nov 03 20:09:52.412035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.411719 5815 scope.go:122] "RemoveContainer" containerID="8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215" Nov 03 20:09:52.421256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.421241 5815 scope.go:122] "RemoveContainer" containerID="8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215" Nov 03 20:09:52.421549 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:09:52.421531 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215\": container with ID starting with 8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215 not found: ID does not exist" containerID="8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215" Nov 03 20:09:52.421618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.421558 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215"} err="failed to get container status \"8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215\": rpc error: code = NotFound desc = could not find container \"8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215\": container with ID starting with 8e81701cad9b45596f863e9f5cefba3ce0a1a89824431697c6c62077ad6c4215 not found: ID does not exist" Nov 03 20:09:52.433017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.432999 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp"] Nov 03 20:09:52.437143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.437121 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-sz7qp"] Nov 03 20:09:52.583568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:09:52.583545 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adad751-6492-456c-869b-731eac0154a9" path="/var/lib/kubelet/pods/7adad751-6492-456c-869b-731eac0154a9/volumes" Nov 03 20:10:13.849876 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:10:13.849842 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:10:43.851148 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:10:43.851066 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:10:47.757882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:10:47.757843 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:10:49.988431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:10:49.988400 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:10:51.745179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:10:51.745148 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:10:54.440022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:10:54.439991 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:10:58.040690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:10:58.040652 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:11:02.077899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:02.077859 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:11:07.671882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:07.671840 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:11:13.850207 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:11:13.850170 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:11:30.756475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:30.756447 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:11:30.762541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:30.762498 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:11:30.787765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:30.787744 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:11:30.793123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:30.793106 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:11:43.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.378851 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77"] Nov 03 20:11:43.379257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.379242 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7adad751-6492-456c-869b-731eac0154a9" containerName="manager" Nov 03 20:11:43.379257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.379254 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adad751-6492-456c-869b-731eac0154a9" containerName="manager" Nov 03 20:11:43.379347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.379341 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adad751-6492-456c-869b-731eac0154a9" containerName="manager" Nov 03 20:11:43.382730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.382705 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.395635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.395175 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77"] Nov 03 20:11:43.517815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.517786 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.517975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.517822 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.517975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.517842 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.517975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.517906 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.517975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.517940 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.517975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.517969 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mck9\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.518139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.518070 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.518139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.518096 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.518200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.518144 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619276 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619307 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619323 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619337 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619469 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619501 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mck9\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619606 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619629 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619807 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619931 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.619979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.619960 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.620208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.620046 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.620208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.620107 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.621892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.621874 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.622042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.622027 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.631587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.631544 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mck9\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.631666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.631647 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-ksr77\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.695707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.695687 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:43.850332 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:11:43.850300 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:11:43.976800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.976774 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77"] Nov 03 20:11:43.978590 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:11:43.978566 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e4bdbb_1ab2_4419_845c_1eb5d9e0dfa6.slice/crio-471d2143297ddc0344d38bb4235f1cfb5feb2a55543cf7e9900abe29580bdd93 WatchSource:0}: Error finding container 471d2143297ddc0344d38bb4235f1cfb5feb2a55543cf7e9900abe29580bdd93: Status 404 returned error can't find the container with id 471d2143297ddc0344d38bb4235f1cfb5feb2a55543cf7e9900abe29580bdd93 Nov 03 20:11:43.980761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.980617 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:43.980761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.980706 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:43.980761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:43.980752 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:44.055689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:44.055661 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" event={"ID":"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6","Type":"ContainerStarted","Data":"0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc"} Nov 03 20:11:44.055804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:44.055737 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" event={"ID":"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6","Type":"ContainerStarted","Data":"471d2143297ddc0344d38bb4235f1cfb5feb2a55543cf7e9900abe29580bdd93"} Nov 03 20:11:44.081173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:44.081111 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podStartSLOduration=1.081093321 podStartE2EDuration="1.081093321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:44.078027506 +0000 UTC m=+613.608373180" watchObservedRunningTime="2026-11-03 20:11:44.081093321 +0000 UTC m=+613.611438995" Nov 03 20:11:44.696524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:44.696467 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:45.698119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:45.698083 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:45.698605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:45.698152 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:46.696718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:46.696678 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:46.696918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:46.696751 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:47.696464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:47.696430 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:47.696888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:47.696489 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:48.696971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:48.696935 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:48.697356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:48.696995 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:49.697116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:49.697081 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:49.697597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:49.697130 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:50.696514 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:50.696477 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:50.696715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:50.696588 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:51.696168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:51.696123 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 20:11:51.696714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:51.696188 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 20:11:52.904024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:52.903989 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:52.904532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:52.904325 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:52.905294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:52.905276 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 20:11:56.272083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:56.272053 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:11:58.497482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.497443 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc"] Nov 03 20:11:58.502315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.502293 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.515891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.515862 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc"] Nov 03 20:11:58.519875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.519856 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8"] Nov 03 20:11:58.523800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.523775 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.552312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.552284 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8"] Nov 03 20:11:58.572199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572166 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572202 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572230 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572279 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572414 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572442 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7dn\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572473 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572605 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572658 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572682 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572756 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572788 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572864 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.572914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572897 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572914 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572933 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.572994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.572950 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhkq\" (UniqueName: \"kubernetes.io/projected/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-kube-api-access-plhkq\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.573156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.573085 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.673892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.673864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.673900 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.673921 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.673941 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhkq\" (UniqueName: \"kubernetes.io/projected/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-kube-api-access-plhkq\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.673969 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674015 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674043 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674084 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674126 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674147 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7dn\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674165 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674207 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674233 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674258 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674288 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674332 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674345 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674370 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.674443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674413 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.675104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674469 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.675104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674704 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.675104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674730 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.675104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674794 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.675104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.674950 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.675104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.675070 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.675730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.675701 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.675977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.675948 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.677473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.677427 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.677801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.677779 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.677872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.677811 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.678317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.678296 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.690232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.690208 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.690332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.690255 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhkq\" (UniqueName: \"kubernetes.io/projected/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-kube-api-access-plhkq\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.690436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.690415 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7dn\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn\") pod \"maas-default-gateway-openshift-default-59f957fdd7-nj4bc\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.690737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.690718 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-gqpt8\" (UID: \"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:58.816464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.816380 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:11:58.849751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:58.849716 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:11:59.108681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.108587 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp"] Nov 03 20:11:59.120558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.113387 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.120855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.120831 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2f29441c-openshift-default-dockercfg-n7nsh\"" Nov 03 20:11:59.121763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.121724 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc"] Nov 03 20:11:59.126848 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:11:59.126814 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8bdf54_ebd5_4b0b_ae0d_fc86f5bb89bf.slice/crio-fa534baa18d0529c210e38e01fa9422f02968137c220981eaa36ff8ea37b239a WatchSource:0}: Error finding container fa534baa18d0529c210e38e01fa9422f02968137c220981eaa36ff8ea37b239a: Status 404 returned error can't find the container with id fa534baa18d0529c210e38e01fa9422f02968137c220981eaa36ff8ea37b239a Nov 03 20:11:59.127821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.127745 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp"] Nov 03 20:11:59.136889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.136855 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.137035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.136941 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.137035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.136973 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.156463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.156438 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8"] Nov 03 20:11:59.159905 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:11:59.159874 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfe5ba3_3ac6_4959_9ee5_6b15ce77aac9.slice/crio-bec8bfd40dee9bd4c2399628b9770cc788227180744661e97490dde7ebc641e2 WatchSource:0}: Error finding container bec8bfd40dee9bd4c2399628b9770cc788227180744661e97490dde7ebc641e2: Status 404 returned error can't find the container with id bec8bfd40dee9bd4c2399628b9770cc788227180744661e97490dde7ebc641e2 Nov 03 20:11:59.164693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.164652 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.164811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.164751 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.164881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.164830 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.173364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.173339 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" event={"ID":"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9","Type":"ContainerStarted","Data":"bec8bfd40dee9bd4c2399628b9770cc788227180744661e97490dde7ebc641e2"} Nov 03 20:11:59.174924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.174894 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" event={"ID":"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf","Type":"ContainerStarted","Data":"fa534baa18d0529c210e38e01fa9422f02968137c220981eaa36ff8ea37b239a"} Nov 03 20:11:59.178987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.178960 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179025 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179098 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76hb5\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179139 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179205 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179229 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179268 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179301 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.179463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.179372 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.280971 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281047 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281076 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76hb5\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281103 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281148 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281175 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281220 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281252 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.281436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.281327 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.282140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.282114 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.284883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.283846 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.284883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.284779 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.285743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.285711 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.285985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.285957 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.286192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.286174 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.286698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.286680 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.296589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.296565 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76hb5\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.298602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.298577 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token\") pod \"e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.442614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.442431 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:11:59.748263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.748223 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp"] Nov 03 20:11:59.760687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.760394 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.760687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.760481 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:11:59.760687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:11:59.760552 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:12:00.214558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.214521 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" event={"ID":"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf","Type":"ContainerStarted","Data":"fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13"} Nov 03 20:12:00.236851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.236673 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" event={"ID":"65989350-ab93-415c-9ae9-6675ec87a182","Type":"ContainerStarted","Data":"d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09"} Nov 03 20:12:00.236851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.236722 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" event={"ID":"65989350-ab93-415c-9ae9-6675ec87a182","Type":"ContainerStarted","Data":"68649347c4f62c7e4dbfde0dd1172602d330eeceab5f2a4ae53883f358b0d0b8"} Nov 03 20:12:00.238459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.238436 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" event={"ID":"ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9","Type":"ContainerStarted","Data":"87faf8984a3b38d53fa53561e53ae78c5e6b60deb1608cfeafe2dfa320a8ca0e"} Nov 03 20:12:00.254352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.254313 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podStartSLOduration=2.254301689 podStartE2EDuration="2.254301689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:00.252828834 +0000 UTC m=+629.783174511" watchObservedRunningTime="2026-11-03 20:12:00.254301689 +0000 UTC m=+629.784647361" Nov 03 20:12:00.287533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.287150 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podStartSLOduration=1.287127725 podStartE2EDuration="1.287127725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:00.286042268 +0000 UTC m=+629.816387941" watchObservedRunningTime="2026-11-03 20:12:00.287127725 +0000 UTC m=+629.817473401" Nov 03 20:12:00.443601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.443564 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:00.448669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.448646 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:00.479634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.479488 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podStartSLOduration=2.479468404 podStartE2EDuration="2.479468404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:00.321732432 +0000 UTC m=+629.852078106" watchObservedRunningTime="2026-11-03 20:12:00.479468404 +0000 UTC m=+630.009814077" Nov 03 20:12:00.825220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.825117 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:12:00.827408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.827374 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:00.827593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.827447 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:00.850204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.850181 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:12:00.851425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.851404 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:00.851601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:00.851574 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:01.244567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:01.244542 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:01.245201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:01.245181 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:01.816968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:01.816927 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:01.817249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:01.817221 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:01.850234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:01.850209 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:01.850604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:01.850255 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:02.826537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:02.825918 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:02.826537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:02.825981 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:02.851120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:02.851087 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:02.851540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:02.851137 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:03.825570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:03.825183 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:03.825570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:03.825271 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:03.850220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:03.850193 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:03.850370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:03.850245 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:04.820450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:04.820417 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:04.821114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:04.821080 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:04.850310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:04.850286 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:04.850521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:04.850482 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:05.817105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:05.817066 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:05.817280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:05.817137 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:05.850395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:05.850364 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:05.850780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:05.850421 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:06.074221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:06.074134 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:06.817518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:06.817464 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 20:12:06.817706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:06.817535 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 20:12:06.850439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:06.850403 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 20:12:06.850953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:06.850464 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 20:12:08.429168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.429134 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:12:08.430270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.430224 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:12:08.431023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.431005 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" Nov 03 20:12:08.445385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.445351 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:12:08.446105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.445938 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:12:08.447154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.447136 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:12:08.781336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.781307 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m"] Nov 03 20:12:08.786173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.786152 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.790362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.790340 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:12:08.794129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.794101 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m"] Nov 03 20:12:08.794251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.794226 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:08.797192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.797167 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.797454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.797429 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5l2\" (UniqueName: \"kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.797605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.797527 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.898480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.898444 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.898659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.898490 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5l2\" (UniqueName: \"kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.898659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.898563 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.899265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.899240 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.901048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.901021 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:08.911092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:08.911065 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5l2\" (UniqueName: \"kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2\") pod \"maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:09.109776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:09.109699 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:09.389439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:09.389412 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m"] Nov 03 20:12:09.391038 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:09.391013 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e88064d_355b_4bc6_9d87_653dae69cb80.slice/crio-fba9f9d31814d1b87cae733a8c64882693fe98da6a93cdc0bf31f2d13f1bbec8 WatchSource:0}: Error finding container fba9f9d31814d1b87cae733a8c64882693fe98da6a93cdc0bf31f2d13f1bbec8: Status 404 returned error can't find the container with id fba9f9d31814d1b87cae733a8c64882693fe98da6a93cdc0bf31f2d13f1bbec8 Nov 03 20:12:09.393016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:09.392995 5815 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:12:10.332398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:10.332357 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" event={"ID":"0e88064d-355b-4bc6-9d87-653dae69cb80","Type":"ContainerStarted","Data":"fba9f9d31814d1b87cae733a8c64882693fe98da6a93cdc0bf31f2d13f1bbec8"} Nov 03 20:12:13.854042 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:12:13.853962 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:12:14.261360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.261332 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:14.381534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.380721 5815 generic.go:356] "Generic (PLEG): container finished" podID="0e88064d-355b-4bc6-9d87-653dae69cb80" containerID="3e7075029dfa4dd4d8f5260aca4be5a1fc8e0c1f802b6d153c2007140ed3fcb4" exitCode=0 Nov 03 20:12:14.381534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.380807 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" event={"ID":"0e88064d-355b-4bc6-9d87-653dae69cb80","Type":"ContainerDied","Data":"3e7075029dfa4dd4d8f5260aca4be5a1fc8e0c1f802b6d153c2007140ed3fcb4"} Nov 03 20:12:14.401914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.401883 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7"] Nov 03 20:12:14.406491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.406467 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.416611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.416498 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7"] Nov 03 20:12:14.445593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.445492 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.445593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.445565 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtwq\" (UniqueName: \"kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.445801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.445643 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.546161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.546129 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.546161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.546167 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtwq\" (UniqueName: \"kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.546407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.546192 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.546943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.546919 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.549023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.549002 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.562946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.562926 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtwq\" (UniqueName: \"kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq\") pod \"maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.715851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.715814 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l"] Nov 03 20:12:14.719723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.719702 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.720883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.720864 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:14.728749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.728728 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l"] Nov 03 20:12:14.747610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.747583 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.747714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.747650 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.747714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.747706 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zx9\" (UniqueName: \"kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.848628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.848591 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.848758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.848664 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.848758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.848696 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zx9\" (UniqueName: \"kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.849210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.849191 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.851369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.851339 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:14.861125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:14.861105 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zx9\" (UniqueName: \"kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9\") pod \"maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:15.028652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.028626 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7"] Nov 03 20:12:15.030531 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:15.030477 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3254e5_efc3_4cef_94b6_564d0925f7ca.slice/crio-b4c339f16d168d390c89143da12ffeac4e598cc2d3878bd2021b32f23a14f710 WatchSource:0}: Error finding container b4c339f16d168d390c89143da12ffeac4e598cc2d3878bd2021b32f23a14f710: Status 404 returned error can't find the container with id b4c339f16d168d390c89143da12ffeac4e598cc2d3878bd2021b32f23a14f710 Nov 03 20:12:15.038646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.038621 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:15.340355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.340330 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l"] Nov 03 20:12:15.342730 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:15.342706 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58eca91f_6d27_44b4_976e_418cc51c9b46.slice/crio-33203b89a3779ef5878a6c4dde3f9881620b020bb1004421bb39d97a3ffcb4aa WatchSource:0}: Error finding container 33203b89a3779ef5878a6c4dde3f9881620b020bb1004421bb39d97a3ffcb4aa: Status 404 returned error can't find the container with id 33203b89a3779ef5878a6c4dde3f9881620b020bb1004421bb39d97a3ffcb4aa Nov 03 20:12:15.398849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.398815 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" event={"ID":"58eca91f-6d27-44b4-976e-418cc51c9b46","Type":"ContainerStarted","Data":"33203b89a3779ef5878a6c4dde3f9881620b020bb1004421bb39d97a3ffcb4aa"} Nov 03 20:12:15.400494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.400468 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" event={"ID":"1b3254e5-efc3-4cef-94b6-564d0925f7ca","Type":"ContainerStarted","Data":"beb22d978f61065f3ea6bc216fba112c56fe3991bd59178e1f3cc554e1400299"} Nov 03 20:12:15.400690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.400562 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" event={"ID":"1b3254e5-efc3-4cef-94b6-564d0925f7ca","Type":"ContainerStarted","Data":"b4c339f16d168d390c89143da12ffeac4e598cc2d3878bd2021b32f23a14f710"} Nov 03 20:12:15.420337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.420277 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" podStartSLOduration=1.420257356 podStartE2EDuration="1.420257356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:15.419156496 +0000 UTC m=+644.949502170" watchObservedRunningTime="2026-11-03 20:12:15.420257356 +0000 UTC m=+644.950603029" Nov 03 20:12:15.632860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.632834 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:15.656434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.656409 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2\" (UniqueName: \"kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2\") pod \"0e88064d-355b-4bc6-9d87-653dae69cb80\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " Nov 03 20:12:15.656614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.656563 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca\") pod \"0e88064d-355b-4bc6-9d87-653dae69cb80\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " Nov 03 20:12:15.656685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.656618 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca\") pod \"0e88064d-355b-4bc6-9d87-653dae69cb80\" (UID: \"0e88064d-355b-4bc6-9d87-653dae69cb80\") " Nov 03 20:12:15.656976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.656947 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca" pod "0e88064d-355b-4bc6-9d87-653dae69cb80" (UID: "0e88064d-355b-4bc6-9d87-653dae69cb80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:15.659055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.659029 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2" pod "0e88064d-355b-4bc6-9d87-653dae69cb80" (UID: "0e88064d-355b-4bc6-9d87-653dae69cb80"). InnerVolumeSpecName "kube-api-access-lh5l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:15.689699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.689594 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca" pod "0e88064d-355b-4bc6-9d87-653dae69cb80" (UID: "0e88064d-355b-4bc6-9d87-653dae69cb80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:15.757807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.757782 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:15.757807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.757804 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e88064d-355b-4bc6-9d87-653dae69cb80-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:15.757954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:15.757816 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5l2\" (UniqueName: \"kubernetes.io/projected/0e88064d-355b-4bc6-9d87-653dae69cb80-kube-api-access-lh5l2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:16.128844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.128813 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986"] Nov 03 20:12:16.129275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.129261 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0e88064d-355b-4bc6-9d87-653dae69cb80" containerName="revoke-keys" Nov 03 20:12:16.129322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.129277 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e88064d-355b-4bc6-9d87-653dae69cb80" containerName="revoke-keys" Nov 03 20:12:16.129385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.129375 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e88064d-355b-4bc6-9d87-653dae69cb80" containerName="revoke-keys" Nov 03 20:12:16.132454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.132439 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.146298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.146272 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986"] Nov 03 20:12:16.161659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.161636 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.161761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.161694 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvdr\" (UniqueName: \"kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.161812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.161772 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.263040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.263010 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.263190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.263066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvdr\" (UniqueName: \"kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.263190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.263097 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.263694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.263671 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.265540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.265498 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.275402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.275379 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvdr\" (UniqueName: \"kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr\") pod \"maas-api-revoke-keys-e2e-ait-4bd752fd-4r986\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.409832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.409809 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" Nov 03 20:12:16.409979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.409849 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m" event={"ID":"0e88064d-355b-4bc6-9d87-653dae69cb80","Type":"ContainerDied","Data":"fba9f9d31814d1b87cae733a8c64882693fe98da6a93cdc0bf31f2d13f1bbec8"} Nov 03 20:12:16.409979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.409905 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fba9f9d31814d1b87cae733a8c64882693fe98da6a93cdc0bf31f2d13f1bbec8" Nov 03 20:12:16.411801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.411778 5815 generic.go:356] "Generic (PLEG): container finished" podID="58eca91f-6d27-44b4-976e-418cc51c9b46" containerID="0c7319a47e94108eab0fa97c4ff1e6c1ed906f47753f3333fbfdf9af1cd3c21e" exitCode=0 Nov 03 20:12:16.411926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.411860 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" event={"ID":"58eca91f-6d27-44b4-976e-418cc51c9b46","Type":"ContainerDied","Data":"0c7319a47e94108eab0fa97c4ff1e6c1ed906f47753f3333fbfdf9af1cd3c21e"} Nov 03 20:12:16.413757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.413731 5815 generic.go:356] "Generic (PLEG): container finished" podID="1b3254e5-efc3-4cef-94b6-564d0925f7ca" containerID="beb22d978f61065f3ea6bc216fba112c56fe3991bd59178e1f3cc554e1400299" exitCode=0 Nov 03 20:12:16.413909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.413839 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" event={"ID":"1b3254e5-efc3-4cef-94b6-564d0925f7ca","Type":"ContainerDied","Data":"beb22d978f61065f3ea6bc216fba112c56fe3991bd59178e1f3cc554e1400299"} Nov 03 20:12:16.451938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.451912 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:16.742114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:16.742042 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986"] Nov 03 20:12:16.744466 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:16.744432 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e6ea2a_496e_4156_8c51_c17ab18c962a.slice/crio-616678f195fd4ee5336dcc9169d51e8468782b08a61c6f2b1db47e57faa68491 WatchSource:0}: Error finding container 616678f195fd4ee5336dcc9169d51e8468782b08a61c6f2b1db47e57faa68491: Status 404 returned error can't find the container with id 616678f195fd4ee5336dcc9169d51e8468782b08a61c6f2b1db47e57faa68491 Nov 03 20:12:17.286727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.286684 5815 patch_prober.go:29] interesting pod/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:12:17.287261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.286757 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:17.423705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.423622 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" event={"ID":"78e6ea2a-496e-4156-8c51-c17ab18c962a","Type":"ContainerStarted","Data":"5f355acf8b142a7368c040aaeb56191ed30d442437c019523aaaa86ab3cea935"} Nov 03 20:12:17.423705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.423676 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" event={"ID":"78e6ea2a-496e-4156-8c51-c17ab18c962a","Type":"ContainerStarted","Data":"616678f195fd4ee5336dcc9169d51e8468782b08a61c6f2b1db47e57faa68491"} Nov 03 20:12:17.443868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.443356 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" podStartSLOduration=1.4433364260000001 podStartE2EDuration="1.443336426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:17.442837531 +0000 UTC m=+646.973183205" watchObservedRunningTime="2026-11-03 20:12:17.443336426 +0000 UTC m=+646.973682098" Nov 03 20:12:17.594535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.594495 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:17.597767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.597746 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:17.679205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.679133 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq\" (UniqueName: \"kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq\") pod \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " Nov 03 20:12:17.679205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.679184 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca\") pod \"58eca91f-6d27-44b4-976e-418cc51c9b46\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " Nov 03 20:12:17.679412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.679233 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9\" (UniqueName: \"kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9\") pod \"58eca91f-6d27-44b4-976e-418cc51c9b46\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " Nov 03 20:12:17.679412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.679251 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca\") pod \"58eca91f-6d27-44b4-976e-418cc51c9b46\" (UID: \"58eca91f-6d27-44b4-976e-418cc51c9b46\") " Nov 03 20:12:17.679412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.679311 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca\") pod \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " Nov 03 20:12:17.679412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.679365 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca\") pod \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\" (UID: \"1b3254e5-efc3-4cef-94b6-564d0925f7ca\") " Nov 03 20:12:17.680481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.680220 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca" pod "58eca91f-6d27-44b4-976e-418cc51c9b46" (UID: "58eca91f-6d27-44b4-976e-418cc51c9b46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:17.680666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.680634 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca" pod "1b3254e5-efc3-4cef-94b6-564d0925f7ca" (UID: "1b3254e5-efc3-4cef-94b6-564d0925f7ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:17.682300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.682274 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca" pod "1b3254e5-efc3-4cef-94b6-564d0925f7ca" (UID: "1b3254e5-efc3-4cef-94b6-564d0925f7ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:17.682401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.682377 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca" pod "58eca91f-6d27-44b4-976e-418cc51c9b46" (UID: "58eca91f-6d27-44b4-976e-418cc51c9b46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:17.682489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.682470 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq" pod "1b3254e5-efc3-4cef-94b6-564d0925f7ca" (UID: "1b3254e5-efc3-4cef-94b6-564d0925f7ca"). InnerVolumeSpecName "kube-api-access-fjtwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:17.682591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.682576 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9" pod "58eca91f-6d27-44b4-976e-418cc51c9b46" (UID: "58eca91f-6d27-44b4-976e-418cc51c9b46"). InnerVolumeSpecName "kube-api-access-h4zx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:17.780992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.780946 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtwq\" (UniqueName: \"kubernetes.io/projected/1b3254e5-efc3-4cef-94b6-564d0925f7ca-kube-api-access-fjtwq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:17.780992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.780994 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:17.781238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.781009 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zx9\" (UniqueName: \"kubernetes.io/projected/58eca91f-6d27-44b4-976e-418cc51c9b46-kube-api-access-h4zx9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:17.781238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.781025 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58eca91f-6d27-44b4-976e-418cc51c9b46-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:17.781238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.781041 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:17.781238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:17.781054 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3254e5-efc3-4cef-94b6-564d0925f7ca-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:18.433537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.433495 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" Nov 03 20:12:18.433537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.433493 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l" event={"ID":"58eca91f-6d27-44b4-976e-418cc51c9b46","Type":"ContainerDied","Data":"33203b89a3779ef5878a6c4dde3f9881620b020bb1004421bb39d97a3ffcb4aa"} Nov 03 20:12:18.433987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.433572 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33203b89a3779ef5878a6c4dde3f9881620b020bb1004421bb39d97a3ffcb4aa" Nov 03 20:12:18.435022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.434996 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" event={"ID":"1b3254e5-efc3-4cef-94b6-564d0925f7ca","Type":"ContainerDied","Data":"b4c339f16d168d390c89143da12ffeac4e598cc2d3878bd2021b32f23a14f710"} Nov 03 20:12:18.435128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.435022 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7" Nov 03 20:12:18.435128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.435042 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c339f16d168d390c89143da12ffeac4e598cc2d3878bd2021b32f23a14f710" Nov 03 20:12:18.436896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.436875 5815 generic.go:356] "Generic (PLEG): container finished" podID="78e6ea2a-496e-4156-8c51-c17ab18c962a" containerID="5f355acf8b142a7368c040aaeb56191ed30d442437c019523aaaa86ab3cea935" exitCode=0 Nov 03 20:12:18.437020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:18.436953 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" event={"ID":"78e6ea2a-496e-4156-8c51-c17ab18c962a","Type":"ContainerDied","Data":"5f355acf8b142a7368c040aaeb56191ed30d442437c019523aaaa86ab3cea935"} Nov 03 20:12:19.444867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.444792 5815 generic.go:356] "Generic (PLEG): container finished" podID="65989350-ab93-415c-9ae9-6675ec87a182" containerID="d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09" exitCode=0 Nov 03 20:12:19.445379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.444864 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" event={"ID":"65989350-ab93-415c-9ae9-6675ec87a182","Type":"ContainerDied","Data":"d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09"} Nov 03 20:12:19.445379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.445228 5815 scope.go:122] "RemoveContainer" containerID="d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09" Nov 03 20:12:19.446466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.446440 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:12:19.446550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.446528 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:12:19.446594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.446569 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:12:19.611880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.611856 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:19.710469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.710439 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca\") pod \"78e6ea2a-496e-4156-8c51-c17ab18c962a\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " Nov 03 20:12:19.710655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.710519 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca\") pod \"78e6ea2a-496e-4156-8c51-c17ab18c962a\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " Nov 03 20:12:19.710655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.710606 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr\" (UniqueName: \"kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr\") pod \"78e6ea2a-496e-4156-8c51-c17ab18c962a\" (UID: \"78e6ea2a-496e-4156-8c51-c17ab18c962a\") " Nov 03 20:12:19.710819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.710796 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca" pod "78e6ea2a-496e-4156-8c51-c17ab18c962a" (UID: "78e6ea2a-496e-4156-8c51-c17ab18c962a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:19.712812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.712781 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca" pod "78e6ea2a-496e-4156-8c51-c17ab18c962a" (UID: "78e6ea2a-496e-4156-8c51-c17ab18c962a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:19.712932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.712832 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr" pod "78e6ea2a-496e-4156-8c51-c17ab18c962a" (UID: "78e6ea2a-496e-4156-8c51-c17ab18c962a"). InnerVolumeSpecName "kube-api-access-7bvdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:19.812195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.812116 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:19.812195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.812147 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e6ea2a-496e-4156-8c51-c17ab18c962a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:19.812195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:19.812160 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvdr\" (UniqueName: \"kubernetes.io/projected/78e6ea2a-496e-4156-8c51-c17ab18c962a-kube-api-access-7bvdr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:20.453925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:20.453887 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" event={"ID":"65989350-ab93-415c-9ae9-6675ec87a182","Type":"ContainerStarted","Data":"faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540"} Nov 03 20:12:20.455595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:20.455562 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" event={"ID":"78e6ea2a-496e-4156-8c51-c17ab18c962a","Type":"ContainerDied","Data":"616678f195fd4ee5336dcc9169d51e8468782b08a61c6f2b1db47e57faa68491"} Nov 03 20:12:20.455720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:20.455614 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="616678f195fd4ee5336dcc9169d51e8468782b08a61c6f2b1db47e57faa68491" Nov 03 20:12:20.455720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:20.455640 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986" Nov 03 20:12:21.443453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.443420 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:21.443675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.443652 5815 patch_prober.go:29] interesting pod/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:12:21.443731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.443715 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 20:12:21.813831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.813798 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d"] Nov 03 20:12:21.814229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814217 5815 cpu_manager.go:414] "containerMap: removing container" podUID="78e6ea2a-496e-4156-8c51-c17ab18c962a" containerName="revoke-keys" Nov 03 20:12:21.814273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814230 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e6ea2a-496e-4156-8c51-c17ab18c962a" containerName="revoke-keys" Nov 03 20:12:21.814273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814247 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1b3254e5-efc3-4cef-94b6-564d0925f7ca" containerName="revoke-keys" Nov 03 20:12:21.814273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814252 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3254e5-efc3-4cef-94b6-564d0925f7ca" containerName="revoke-keys" Nov 03 20:12:21.814273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814270 5815 cpu_manager.go:414] "containerMap: removing container" podUID="58eca91f-6d27-44b4-976e-418cc51c9b46" containerName="revoke-keys" Nov 03 20:12:21.814398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814275 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58eca91f-6d27-44b4-976e-418cc51c9b46" containerName="revoke-keys" Nov 03 20:12:21.814398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814357 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e6ea2a-496e-4156-8c51-c17ab18c962a" containerName="revoke-keys" Nov 03 20:12:21.814398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814367 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3254e5-efc3-4cef-94b6-564d0925f7ca" containerName="revoke-keys" Nov 03 20:12:21.814398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.814375 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="58eca91f-6d27-44b4-976e-418cc51c9b46" containerName="revoke-keys" Nov 03 20:12:21.817414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.817400 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:21.825388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.825368 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:21.825527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.825483 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:12:21.826760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.826740 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d"] Nov 03 20:12:21.931063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.931034 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7n2\" (UniqueName: \"kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:21.931282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.931099 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:21.931282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:21.931213 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.032601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.032562 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.032800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.032615 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.032800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.032673 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7n2\" (UniqueName: \"kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.033220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.033197 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.035132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.035110 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.045179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.045156 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7n2\" (UniqueName: \"kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2\") pod \"maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.132244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.132177 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:22.427936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.427913 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d"] Nov 03 20:12:22.429983 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:22.429950 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c701ca_8b58_477f_bef0_d6b2e57916fb.slice/crio-42703305e6462fc707b4cfc6d041685e912cac7832e22a9d933a1f2e4aaf9a89 WatchSource:0}: Error finding container 42703305e6462fc707b4cfc6d041685e912cac7832e22a9d933a1f2e4aaf9a89: Status 404 returned error can't find the container with id 42703305e6462fc707b4cfc6d041685e912cac7832e22a9d933a1f2e4aaf9a89 Nov 03 20:12:22.450549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.446561 5815 patch_prober.go:29] interesting pod/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:12:22.450549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.446612 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 20:12:22.509574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.508990 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" event={"ID":"54c701ca-8b58-477f-bef0-d6b2e57916fb","Type":"ContainerStarted","Data":"42703305e6462fc707b4cfc6d041685e912cac7832e22a9d933a1f2e4aaf9a89"} Nov 03 20:12:22.810940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.810910 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m"] Nov 03 20:12:22.818581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:22.818560 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-459ca04a-qzx2m"] Nov 03 20:12:23.443393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:23.443357 5815 patch_prober.go:29] interesting pod/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:12:23.443627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:23.443411 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 20:12:23.516639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:23.516608 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" event={"ID":"54c701ca-8b58-477f-bef0-d6b2e57916fb","Type":"ContainerStarted","Data":"5ec0fa2b77fbea08fecbee3f93744dcae456d7fc930a66b9709ce64a1dbaff07"} Nov 03 20:12:23.536164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:23.536112 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" podStartSLOduration=2.536098586 podStartE2EDuration="2.536098586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:23.534838882 +0000 UTC m=+653.065184555" watchObservedRunningTime="2026-11-03 20:12:23.536098586 +0000 UTC m=+653.066444261" Nov 03 20:12:24.443346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:24.443315 5815 patch_prober.go:29] interesting pod/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:12:24.443818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:24.443366 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 20:12:24.525497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:24.525466 5815 generic.go:356] "Generic (PLEG): container finished" podID="54c701ca-8b58-477f-bef0-d6b2e57916fb" containerID="5ec0fa2b77fbea08fecbee3f93744dcae456d7fc930a66b9709ce64a1dbaff07" exitCode=0 Nov 03 20:12:24.525686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:24.525562 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" event={"ID":"54c701ca-8b58-477f-bef0-d6b2e57916fb","Type":"ContainerDied","Data":"5ec0fa2b77fbea08fecbee3f93744dcae456d7fc930a66b9709ce64a1dbaff07"} Nov 03 20:12:24.625609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:24.625573 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e88064d-355b-4bc6-9d87-653dae69cb80" path="/var/lib/kubelet/pods/0e88064d-355b-4bc6-9d87-653dae69cb80/volumes" Nov 03 20:12:25.199040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.198964 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l"] Nov 03 20:12:25.201933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.201906 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9dd42e13-75h5l"] Nov 03 20:12:25.443444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.443401 5815 patch_prober.go:29] interesting pod/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 03 20:12:25.443991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.443461 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 03 20:12:25.708893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.708868 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:25.804795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.804713 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca\") pod \"54c701ca-8b58-477f-bef0-d6b2e57916fb\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " Nov 03 20:12:25.804939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.804811 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2\" (UniqueName: \"kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2\") pod \"54c701ca-8b58-477f-bef0-d6b2e57916fb\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " Nov 03 20:12:25.804939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.804860 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca\") pod \"54c701ca-8b58-477f-bef0-d6b2e57916fb\" (UID: \"54c701ca-8b58-477f-bef0-d6b2e57916fb\") " Nov 03 20:12:25.805159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.805129 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca" pod "54c701ca-8b58-477f-bef0-d6b2e57916fb" (UID: "54c701ca-8b58-477f-bef0-d6b2e57916fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:25.807045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.807021 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca" pod "54c701ca-8b58-477f-bef0-d6b2e57916fb" (UID: "54c701ca-8b58-477f-bef0-d6b2e57916fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:25.807206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.807182 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2" pod "54c701ca-8b58-477f-bef0-d6b2e57916fb" (UID: "54c701ca-8b58-477f-bef0-d6b2e57916fb"). InnerVolumeSpecName "kube-api-access-2b7n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:25.876371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.876338 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:25.906478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.906449 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.906649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.906481 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7n2\" (UniqueName: \"kubernetes.io/projected/54c701ca-8b58-477f-bef0-d6b2e57916fb-kube-api-access-2b7n2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.906649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:25.906497 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54c701ca-8b58-477f-bef0-d6b2e57916fb-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:26.446861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.446825 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:26.447344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.447216 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:26.447883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.447867 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:26.567418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.567340 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" event={"ID":"54c701ca-8b58-477f-bef0-d6b2e57916fb","Type":"ContainerDied","Data":"42703305e6462fc707b4cfc6d041685e912cac7832e22a9d933a1f2e4aaf9a89"} Nov 03 20:12:26.567698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.567677 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42703305e6462fc707b4cfc6d041685e912cac7832e22a9d933a1f2e4aaf9a89" Nov 03 20:12:26.568183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.568162 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d" Nov 03 20:12:26.585647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:26.585622 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58eca91f-6d27-44b4-976e-418cc51c9b46" path="/var/lib/kubelet/pods/58eca91f-6d27-44b4-976e-418cc51c9b46/volumes" Nov 03 20:12:27.269778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:27.269743 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986"] Nov 03 20:12:27.272688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:27.272664 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd752fd-4r986"] Nov 03 20:12:27.480907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:27.480875 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7"] Nov 03 20:12:27.483553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:27.483527 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e17fc195-4v6m7"] Nov 03 20:12:28.585653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:28.585622 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3254e5-efc3-4cef-94b6-564d0925f7ca" path="/var/lib/kubelet/pods/1b3254e5-efc3-4cef-94b6-564d0925f7ca/volumes" Nov 03 20:12:28.586019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:28.586004 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e6ea2a-496e-4156-8c51-c17ab18c962a" path="/var/lib/kubelet/pods/78e6ea2a-496e-4156-8c51-c17ab18c962a/volumes" Nov 03 20:12:32.721569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:32.721529 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d"] Nov 03 20:12:32.724654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:32.724629 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f29441c-sfg4d"] Nov 03 20:12:32.910308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:32.910277 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp"] Nov 03 20:12:32.910554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:32.910490 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" containerID="cri-o://faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540" gracePeriod=30 Nov 03 20:12:34.319086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:34.319049 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:34.586988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:34.586891 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c701ca-8b58-477f-bef0-d6b2e57916fb" path="/var/lib/kubelet/pods/54c701ca-8b58-477f-bef0-d6b2e57916fb/volumes" Nov 03 20:12:38.167705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.167682 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:38.338862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.338825 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.339143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.339125 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.339937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.339128 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:38.339937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.339307 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:38.340184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.340169 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.340591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.340564 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.343307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.343277 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.343931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.343806 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:38.344373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.344356 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.344593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.344577 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.345525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.345456 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:38.346108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.345701 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.346108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.345775 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket\") pod \"65989350-ab93-415c-9ae9-6675ec87a182\" (UID: \"65989350-ab93-415c-9ae9-6675ec87a182\") " Nov 03 20:12:38.346108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.346072 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:38.346318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.346190 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65989350-ab93-415c-9ae9-6675ec87a182-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.346318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.346210 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.346318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.346227 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.346318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.346241 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.346318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.346256 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.350793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.350745 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:38.359125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.358930 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:38.359352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.359331 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "kube-api-access-76hb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:38.360632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.360605 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo" pod "65989350-ab93-415c-9ae9-6675ec87a182" (UID: "65989350-ab93-415c-9ae9-6675ec87a182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:12:38.447050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.447019 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65989350-ab93-415c-9ae9-6675ec87a182-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.447050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.447049 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76hb5\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-kube-api-access-76hb5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.447238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.447060 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65989350-ab93-415c-9ae9-6675ec87a182-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.447238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.447070 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65989350-ab93-415c-9ae9-6675ec87a182-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:38.670107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.670069 5815 generic.go:356] "Generic (PLEG): container finished" podID="65989350-ab93-415c-9ae9-6675ec87a182" containerID="faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540" exitCode=0 Nov 03 20:12:38.670297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.670119 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" Nov 03 20:12:38.670297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.670156 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" event={"ID":"65989350-ab93-415c-9ae9-6675ec87a182","Type":"ContainerDied","Data":"faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540"} Nov 03 20:12:38.670297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.670242 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp" event={"ID":"65989350-ab93-415c-9ae9-6675ec87a182","Type":"ContainerDied","Data":"68649347c4f62c7e4dbfde0dd1172602d330eeceab5f2a4ae53883f358b0d0b8"} Nov 03 20:12:38.670297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.670272 5815 scope.go:122] "RemoveContainer" containerID="faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540" Nov 03 20:12:38.680453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.680436 5815 scope.go:122] "RemoveContainer" containerID="d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09" Nov 03 20:12:38.689460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.689443 5815 scope.go:122] "RemoveContainer" containerID="faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540" Nov 03 20:12:38.689761 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:12:38.689740 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540\": container with ID starting with faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540 not found: ID does not exist" containerID="faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540" Nov 03 20:12:38.689841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.689769 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540"} err="failed to get container status \"faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540\": rpc error: code = NotFound desc = could not find container \"faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540\": container with ID starting with faed008a608a4f5c15b0b7031ac9d34a4257be74919fc3ce897e1624934b0540 not found: ID does not exist" Nov 03 20:12:38.689841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.689785 5815 scope.go:122] "RemoveContainer" containerID="d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09" Nov 03 20:12:38.690050 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:12:38.690034 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09\": container with ID starting with d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09 not found: ID does not exist" containerID="d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09" Nov 03 20:12:38.690108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.690055 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09"} err="failed to get container status \"d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09\": rpc error: code = NotFound desc = could not find container \"d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09\": container with ID starting with d6cbf8c9c470a854f4733b620916840fbc032382359e6318c0d0f0e9516c2f09 not found: ID does not exist" Nov 03 20:12:38.695899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.695880 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp"] Nov 03 20:12:38.711838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:38.711820 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2f29441c-openshift-default-7f78d7dbdf-xhhdp"] Nov 03 20:12:40.584244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:40.584217 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65989350-ab93-415c-9ae9-6675ec87a182" path="/var/lib/kubelet/pods/65989350-ab93-415c-9ae9-6675ec87a182/volumes" Nov 03 20:12:42.657497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:42.657462 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:43.852366 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:12:43.852334 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:12:46.042167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042138 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2"] Nov 03 20:12:46.042575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042551 5815 cpu_manager.go:414] "containerMap: removing container" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" Nov 03 20:12:46.042575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042564 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" Nov 03 20:12:46.042667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042593 5815 cpu_manager.go:414] "containerMap: removing container" podUID="54c701ca-8b58-477f-bef0-d6b2e57916fb" containerName="revoke-keys" Nov 03 20:12:46.042667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042598 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c701ca-8b58-477f-bef0-d6b2e57916fb" containerName="revoke-keys" Nov 03 20:12:46.042736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042668 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" Nov 03 20:12:46.042736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042677 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" Nov 03 20:12:46.042736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.042687 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c701ca-8b58-477f-bef0-d6b2e57916fb" containerName="revoke-keys" Nov 03 20:12:46.046713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.046690 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.058121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.058099 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"openshift-service-ca.crt\"" Nov 03 20:12:46.059628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.059606 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"simulator-w3-4f93d1-kserve-self-signed-certs\"" Nov 03 20:12:46.059765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.059671 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"kube-root-ca.crt\"" Nov 03 20:12:46.059830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.059696 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"default-dockercfg-qk5xs\"" Nov 03 20:12:46.061661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.061639 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2"] Nov 03 20:12:46.121616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.121574 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.121777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.121633 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.121821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.121789 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxd4s\" (UniqueName: \"kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.121854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.121829 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.121940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.121921 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.122023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.122002 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223026 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223074 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223139 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223172 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223241 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxd4s\" (UniqueName: \"kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223274 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223762 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.223992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223968 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.224099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.223998 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.227254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.227232 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.239875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.239843 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxd4s\" (UniqueName: \"kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.244299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.244222 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm\") pod \"simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.378922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.378836 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:46.668558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.668528 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2"] Nov 03 20:12:46.670225 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:46.670192 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3870873f_d57a_4a3b_b82a_25d596f20c80.slice/crio-d8ffcddfa8f4b1ed5a27051b2b62acd443f5f73d9a7033996df6ef2d61629f22 WatchSource:0}: Error finding container d8ffcddfa8f4b1ed5a27051b2b62acd443f5f73d9a7033996df6ef2d61629f22: Status 404 returned error can't find the container with id d8ffcddfa8f4b1ed5a27051b2b62acd443f5f73d9a7033996df6ef2d61629f22 Nov 03 20:12:46.726467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:46.726435 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" event={"ID":"3870873f-d57a-4a3b-b82a-25d596f20c80","Type":"ContainerStarted","Data":"d8ffcddfa8f4b1ed5a27051b2b62acd443f5f73d9a7033996df6ef2d61629f22"} Nov 03 20:12:48.055492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:48.055460 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:48.751046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:48.750998 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" event={"ID":"3870873f-d57a-4a3b-b82a-25d596f20c80","Type":"ContainerStarted","Data":"b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e"} Nov 03 20:12:48.772728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:48.772684 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podStartSLOduration=1.446757995 podStartE2EDuration="2.772670558s" totalImagesPullingTime="1.325912563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:48.77035847 +0000 UTC m=+678.300704141" watchObservedRunningTime="2026-11-03 20:12:48.772670558 +0000 UTC m=+678.303016230" Nov 03 20:12:49.379224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:49.379190 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:50.973235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.973201 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq"] Nov 03 20:12:50.973837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.973818 5815 cpu_manager.go:414] "containerMap: removing container" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" Nov 03 20:12:50.973878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.973841 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65989350-ab93-415c-9ae9-6675ec87a182" containerName="istio-proxy" Nov 03 20:12:50.977717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.977699 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:50.982934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.982894 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:12:50.982934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.982911 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:12:50.982934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.982911 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:12:50.986274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.986239 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kbftm\"" Nov 03 20:12:50.989968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:50.989948 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq"] Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.070730 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjf6s\" (UniqueName: \"kubernetes.io/projected/b69e4a59-126c-4592-b7f5-b2ccf31781d7-kube-api-access-rjf6s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.070790 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.070848 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.070924 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.071121 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69e4a59-126c-4592-b7f5-b2ccf31781d7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.071158 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.072256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.071242 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172316 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172369 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69e4a59-126c-4592-b7f5-b2ccf31781d7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172389 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172442 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172464 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjf6s\" (UniqueName: \"kubernetes.io/projected/b69e4a59-126c-4592-b7f5-b2ccf31781d7-kube-api-access-rjf6s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172488 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.172667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.172561 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.173150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.173124 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.175115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.175082 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69e4a59-126c-4592-b7f5-b2ccf31781d7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.175115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.175105 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.175281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.175217 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.175463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.175446 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.184588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.184563 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69e4a59-126c-4592-b7f5-b2ccf31781d7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.184999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.184979 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjf6s\" (UniqueName: \"kubernetes.io/projected/b69e4a59-126c-4592-b7f5-b2ccf31781d7-kube-api-access-rjf6s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x96lq\" (UID: \"b69e4a59-126c-4592-b7f5-b2ccf31781d7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.292684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.292613 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:51.649870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.649844 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq"] Nov 03 20:12:51.651749 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:12:51.651715 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69e4a59_126c_4592_b7f5_b2ccf31781d7.slice/crio-1c33d6506db0606d0ebe6fb33aa28c7d21d11c44816829887f8c6dfcea8d7380 WatchSource:0}: Error finding container 1c33d6506db0606d0ebe6fb33aa28c7d21d11c44816829887f8c6dfcea8d7380: Status 404 returned error can't find the container with id 1c33d6506db0606d0ebe6fb33aa28c7d21d11c44816829887f8c6dfcea8d7380 Nov 03 20:12:51.769102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:51.769064 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" event={"ID":"b69e4a59-126c-4592-b7f5-b2ccf31781d7","Type":"ContainerStarted","Data":"1c33d6506db0606d0ebe6fb33aa28c7d21d11c44816829887f8c6dfcea8d7380"} Nov 03 20:12:56.379644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:56.379614 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:56.394844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:56.394812 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:56.834238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:56.834208 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:12:57.142580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:57.142481 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:12:57.405265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:57.405232 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:12:57.405706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:57.405295 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:12:57.842528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:57.841652 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" event={"ID":"b69e4a59-126c-4592-b7f5-b2ccf31781d7","Type":"ContainerStarted","Data":"92ddfaf3e1edb9fd94cbfdda3c3b7db06bb5d340f904c8fb3b02fcb9b0cfcb7d"} Nov 03 20:12:57.870709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:57.870651 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" podStartSLOduration=2.120484913 podStartE2EDuration="7.870636254s" totalImagesPullingTime="5.750151341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:57.869881498 +0000 UTC m=+687.400227171" watchObservedRunningTime="2026-11-03 20:12:57.870636254 +0000 UTC m=+687.400981925" Nov 03 20:12:58.847183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:58.847152 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:12:58.849088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:12:58.849065 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x96lq" Nov 03 20:13:07.779658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:07.779624 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:13:13.852485 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:13:13.852448 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:13:15.913492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.913450 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns"] Nov 03 20:13:15.917734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.917715 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:15.923459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.923435 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"openshift-service-ca.crt\"" Nov 03 20:13:15.923616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.923561 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"kube-root-ca.crt\"" Nov 03 20:13:15.923687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.923634 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"premium-w4-466f83-kserve-self-signed-certs\"" Nov 03 20:13:15.923754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.923690 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"default-dockercfg-4lmjg\"" Nov 03 20:13:15.926646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:15.926459 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns"] Nov 03 20:13:16.033629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.033592 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.033829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.033643 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.033829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.033784 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.033951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.033831 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m59dq\" (UniqueName: \"kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.033951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.033920 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.034103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.034077 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.134822 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m59dq\" (UniqueName: \"kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.134893 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.134964 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.135022 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.135048 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.135100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.135360 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.135591 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.141115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.135711 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.152983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.148353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.152983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.150808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.153561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.153498 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m59dq\" (UniqueName: \"kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq\") pod \"premium-w4-466f83-kserve-5d655b97bc-dmpns\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.232374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.232345 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:16.518264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.518240 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns"] Nov 03 20:13:16.521239 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:13:16.521202 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf361fa_7064_41ed_af8d_c674ae0eb7d1.slice/crio-d57e11e2fa541b20c491c3f1f7c749c98a54764421e22001198610f215a742d2 WatchSource:0}: Error finding container d57e11e2fa541b20c491c3f1f7c749c98a54764421e22001198610f215a742d2: Status 404 returned error can't find the container with id d57e11e2fa541b20c491c3f1f7c749c98a54764421e22001198610f215a742d2 Nov 03 20:13:16.974146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.974108 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" event={"ID":"baf361fa-7064-41ed-af8d-c674ae0eb7d1","Type":"ContainerStarted","Data":"6ba19fc6197a89656f8707a3c37de0b06ac4973d68593fedaae2d7434e2e19b5"} Nov 03 20:13:16.974546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.974155 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" event={"ID":"baf361fa-7064-41ed-af8d-c674ae0eb7d1","Type":"ContainerStarted","Data":"d57e11e2fa541b20c491c3f1f7c749c98a54764421e22001198610f215a742d2"} Nov 03 20:13:16.994097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:16.994047 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podStartSLOduration=1.994031035 podStartE2EDuration="1.994031035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:16.992757194 +0000 UTC m=+706.523102866" watchObservedRunningTime="2026-11-03 20:13:16.994031035 +0000 UTC m=+706.524376746" Nov 03 20:13:17.233282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:17.233220 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:20.590195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:20.590165 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:13:26.233567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:26.233530 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:26.246409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:26.246386 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:27.066165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:27.066138 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:13:32.663329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.663299 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd"] Nov 03 20:13:32.668495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.668472 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.682915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.682889 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd"] Nov 03 20:13:32.683232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.683213 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"default-dockercfg-tvmvj\"" Nov 03 20:13:32.683327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.683301 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"openshift-service-ca.crt\"" Nov 03 20:13:32.683393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.683377 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"kube-root-ca.crt\"" Nov 03 20:13:32.683449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.683407 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"premium-w1-b13a14-kserve-self-signed-certs\"" Nov 03 20:13:32.705276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.705248 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.705427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.705307 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.705427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.705407 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.705627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.705458 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.705627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.705560 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7k9h\" (UniqueName: \"kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.705627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.705606 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.807276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807238 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.807540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807309 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.807540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807367 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.807540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807403 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.807540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807463 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7k9h\" (UniqueName: \"kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.807540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807523 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.808010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.807970 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.808108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.808038 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.808259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.808237 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.815580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.815556 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.825647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.823535 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.826061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.825996 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7k9h\" (UniqueName: \"kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h\") pod \"premium-w1-b13a14-kserve-7d457dbbd9-m24pd\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:32.989597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:32.989566 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:33.273441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:33.273395 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd"] Nov 03 20:13:33.274669 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:13:33.274640 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd87d17d2_6ce0_4368_b147_401e31d32450.slice/crio-dc4e5082874719d5c7abc1c1038642ff86e66a2848051da4454649442aec0120 WatchSource:0}: Error finding container dc4e5082874719d5c7abc1c1038642ff86e66a2848051da4454649442aec0120: Status 404 returned error can't find the container with id dc4e5082874719d5c7abc1c1038642ff86e66a2848051da4454649442aec0120 Nov 03 20:13:34.109262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:34.109220 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" event={"ID":"d87d17d2-6ce0-4368-b147-401e31d32450","Type":"ContainerStarted","Data":"eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88"} Nov 03 20:13:34.109262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:34.109268 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" event={"ID":"d87d17d2-6ce0-4368-b147-401e31d32450","Type":"ContainerStarted","Data":"dc4e5082874719d5c7abc1c1038642ff86e66a2848051da4454649442aec0120"} Nov 03 20:13:34.128720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:34.128641 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podStartSLOduration=2.128624928 podStartE2EDuration="2.128624928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:34.127983013 +0000 UTC m=+723.658328686" watchObservedRunningTime="2026-11-03 20:13:34.128624928 +0000 UTC m=+723.658970601" Nov 03 20:13:34.157260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:34.157233 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:13:34.990311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:34.990282 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:42.990622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:42.990394 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:43.008954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:43.008929 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:43.184574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:43.184540 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:13:43.853454 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:13:43.853411 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:13:51.476685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:51.476649 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:13:52.311459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.311423 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm"] Nov 03 20:13:52.316286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.316262 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.329783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.329759 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:52.329962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.329794 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:13:52.336911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.336882 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm"] Nov 03 20:13:52.415593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.415564 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.415748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.415702 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.415748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.415735 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnp8k\" (UniqueName: \"kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.516708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.516673 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.517182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.516719 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnp8k\" (UniqueName: \"kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.517182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.516780 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.517534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.517494 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.519488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.519461 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.529753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.529726 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnp8k\" (UniqueName: \"kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k\") pod \"maas-api-revoke-sub-60164d7c-w4ppm\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.637236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.637162 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:52.951345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:52.951313 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm"] Nov 03 20:13:52.952777 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:13:52.952739 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5388799_7784_418d_9e0b_eaf69c8f2bb8.slice/crio-a333574a1a8139f74eb647f37013e2a641795bd83c62eea9e7ef3a12b358c630 WatchSource:0}: Error finding container a333574a1a8139f74eb647f37013e2a641795bd83c62eea9e7ef3a12b358c630: Status 404 returned error can't find the container with id a333574a1a8139f74eb647f37013e2a641795bd83c62eea9e7ef3a12b358c630 Nov 03 20:13:53.238616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:53.238584 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" event={"ID":"a5388799-7784-418d-9e0b-eaf69c8f2bb8","Type":"ContainerStarted","Data":"a333574a1a8139f74eb647f37013e2a641795bd83c62eea9e7ef3a12b358c630"} Nov 03 20:13:54.246061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:54.246027 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5388799-7784-418d-9e0b-eaf69c8f2bb8" containerID="8a0e7229741da64e2bec3be0e142d95ac087555cd30f578d30b6ea72506df97b" exitCode=0 Nov 03 20:13:54.246443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:54.246117 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" event={"ID":"a5388799-7784-418d-9e0b-eaf69c8f2bb8","Type":"ContainerDied","Data":"8a0e7229741da64e2bec3be0e142d95ac087555cd30f578d30b6ea72506df97b"} Nov 03 20:13:55.396061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.396037 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:55.550449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.550372 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k\" (UniqueName: \"kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k\") pod \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " Nov 03 20:13:55.550635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.550470 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca\") pod \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " Nov 03 20:13:55.550635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.550564 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca\") pod \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\" (UID: \"a5388799-7784-418d-9e0b-eaf69c8f2bb8\") " Nov 03 20:13:55.550908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.550876 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca" pod "a5388799-7784-418d-9e0b-eaf69c8f2bb8" (UID: "a5388799-7784-418d-9e0b-eaf69c8f2bb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:55.552715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.552686 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca" pod "a5388799-7784-418d-9e0b-eaf69c8f2bb8" (UID: "a5388799-7784-418d-9e0b-eaf69c8f2bb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:55.552715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.552687 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k" pod "a5388799-7784-418d-9e0b-eaf69c8f2bb8" (UID: "a5388799-7784-418d-9e0b-eaf69c8f2bb8"). InnerVolumeSpecName "kube-api-access-lnp8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:55.651620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.651593 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnp8k\" (UniqueName: \"kubernetes.io/projected/a5388799-7784-418d-9e0b-eaf69c8f2bb8-kube-api-access-lnp8k\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:55.651620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.651618 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:55.651792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:55.651627 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5388799-7784-418d-9e0b-eaf69c8f2bb8-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:56.263330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:56.263301 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" Nov 03 20:13:56.263559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:56.263383 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm" event={"ID":"a5388799-7784-418d-9e0b-eaf69c8f2bb8","Type":"ContainerDied","Data":"a333574a1a8139f74eb647f37013e2a641795bd83c62eea9e7ef3a12b358c630"} Nov 03 20:13:56.263559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:56.263464 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a333574a1a8139f74eb647f37013e2a641795bd83c62eea9e7ef3a12b358c630" Nov 03 20:13:59.812285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.812246 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x"] Nov 03 20:13:59.812975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.812959 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a5388799-7784-418d-9e0b-eaf69c8f2bb8" containerName="revoke-keys" Nov 03 20:13:59.813034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.812979 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5388799-7784-418d-9e0b-eaf69c8f2bb8" containerName="revoke-keys" Nov 03 20:13:59.813123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.813100 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5388799-7784-418d-9e0b-eaf69c8f2bb8" containerName="revoke-keys" Nov 03 20:13:59.817081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.817064 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:13:59.821265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.821239 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:59.822861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.822835 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:13:59.826811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.826790 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x"] Nov 03 20:13:59.993477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.993447 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:13:59.993665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.993497 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78lp\" (UniqueName: \"kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:13:59.993665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:13:59.993632 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.094721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.094643 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.094721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.094698 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78lp\" (UniqueName: \"kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.094929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.094754 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.095909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.095884 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.101842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.101810 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.108556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.108529 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78lp\" (UniqueName: \"kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp\") pod \"maas-api-revoke-sub-1438dd0d-mpz5x\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.133977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.133952 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:00.412672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:00.412650 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x"] Nov 03 20:14:00.415146 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:00.415108 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807ac60d_de18_4ab4_bd5d_27e64229f29e.slice/crio-c6c3c4cda352451414d9aabcc4fe2c11190d11dea8204ac7885f2f7c36f0b88b WatchSource:0}: Error finding container c6c3c4cda352451414d9aabcc4fe2c11190d11dea8204ac7885f2f7c36f0b88b: Status 404 returned error can't find the container with id c6c3c4cda352451414d9aabcc4fe2c11190d11dea8204ac7885f2f7c36f0b88b Nov 03 20:14:01.303801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:01.303771 5815 generic.go:356] "Generic (PLEG): container finished" podID="807ac60d-de18-4ab4-bd5d-27e64229f29e" containerID="ca4dec2cbe718b4111332de76f15c22510a291b7ab6f0ef43f9df799177039d5" exitCode=0 Nov 03 20:14:01.304166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:01.303860 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" event={"ID":"807ac60d-de18-4ab4-bd5d-27e64229f29e","Type":"ContainerDied","Data":"ca4dec2cbe718b4111332de76f15c22510a291b7ab6f0ef43f9df799177039d5"} Nov 03 20:14:01.304166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:01.303917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" event={"ID":"807ac60d-de18-4ab4-bd5d-27e64229f29e","Type":"ContainerStarted","Data":"c6c3c4cda352451414d9aabcc4fe2c11190d11dea8204ac7885f2f7c36f0b88b"} Nov 03 20:14:02.451196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.451170 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:02.615712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.615638 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm"] Nov 03 20:14:02.616167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.616152 5815 cpu_manager.go:414] "containerMap: removing container" podUID="807ac60d-de18-4ab4-bd5d-27e64229f29e" containerName="revoke-keys" Nov 03 20:14:02.616213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.616169 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807ac60d-de18-4ab4-bd5d-27e64229f29e" containerName="revoke-keys" Nov 03 20:14:02.616297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.616286 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="807ac60d-de18-4ab4-bd5d-27e64229f29e" containerName="revoke-keys" Nov 03 20:14:02.619531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.619498 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.621757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.621736 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca\") pod \"807ac60d-de18-4ab4-bd5d-27e64229f29e\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " Nov 03 20:14:02.621892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.621875 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca\") pod \"807ac60d-de18-4ab4-bd5d-27e64229f29e\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " Nov 03 20:14:02.621938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.621914 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp\" (UniqueName: \"kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp\") pod \"807ac60d-de18-4ab4-bd5d-27e64229f29e\" (UID: \"807ac60d-de18-4ab4-bd5d-27e64229f29e\") " Nov 03 20:14:02.622146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.622119 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca" pod "807ac60d-de18-4ab4-bd5d-27e64229f29e" (UID: "807ac60d-de18-4ab4-bd5d-27e64229f29e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:02.622270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.622252 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:02.624447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.624425 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca" pod "807ac60d-de18-4ab4-bd5d-27e64229f29e" (UID: "807ac60d-de18-4ab4-bd5d-27e64229f29e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:02.624844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.624816 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp" pod "807ac60d-de18-4ab4-bd5d-27e64229f29e" (UID: "807ac60d-de18-4ab4-bd5d-27e64229f29e"). InnerVolumeSpecName "kube-api-access-s78lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:02.629862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.629836 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm"] Nov 03 20:14:02.723170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.723141 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.723302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.723178 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmzl\" (UniqueName: \"kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.723377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.723352 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.723450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.723435 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807ac60d-de18-4ab4-bd5d-27e64229f29e-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:02.723534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.723456 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78lp\" (UniqueName: \"kubernetes.io/projected/807ac60d-de18-4ab4-bd5d-27e64229f29e-kube-api-access-s78lp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:02.824674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.824647 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmzl\" (UniqueName: \"kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.824811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.824715 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.824881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.824838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.825609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.825585 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.827216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.827192 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.842902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.842883 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmzl\" (UniqueName: \"kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl\") pod \"maas-api-revoke-sub-01085c78-rlglm\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:02.951255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:02.951189 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:03.231192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.231171 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm"] Nov 03 20:14:03.233039 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:03.233014 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f85b59_de69_4aed_b299_f35795ff644a.slice/crio-4480a56718c85ea9501b0e258512101039c3cf504b7cb33ea8241ae9aa7f179c WatchSource:0}: Error finding container 4480a56718c85ea9501b0e258512101039c3cf504b7cb33ea8241ae9aa7f179c: Status 404 returned error can't find the container with id 4480a56718c85ea9501b0e258512101039c3cf504b7cb33ea8241ae9aa7f179c Nov 03 20:14:03.317172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.317145 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" event={"ID":"d7f85b59-de69-4aed-b299-f35795ff644a","Type":"ContainerStarted","Data":"4480a56718c85ea9501b0e258512101039c3cf504b7cb33ea8241ae9aa7f179c"} Nov 03 20:14:03.318576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.318556 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" Nov 03 20:14:03.318576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.318564 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x" event={"ID":"807ac60d-de18-4ab4-bd5d-27e64229f29e","Type":"ContainerDied","Data":"c6c3c4cda352451414d9aabcc4fe2c11190d11dea8204ac7885f2f7c36f0b88b"} Nov 03 20:14:03.318715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.318611 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6c3c4cda352451414d9aabcc4fe2c11190d11dea8204ac7885f2f7c36f0b88b" Nov 03 20:14:03.408162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.408138 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8"] Nov 03 20:14:03.413241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.413224 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.422113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.422085 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8"] Nov 03 20:14:03.534435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.534369 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7q8\" (UniqueName: \"kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.534435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.534419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.534818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.534494 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.636051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.636012 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7q8\" (UniqueName: \"kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.636221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.636085 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.636221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.636117 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.636851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.636825 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.638702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.638681 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.650802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.650781 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7q8\" (UniqueName: \"kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8\") pod \"maas-api-revoke-sub-86ef4153-bcvq8\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:03.728204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:03.728177 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:04.017755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.017731 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8"] Nov 03 20:14:04.019567 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:04.019544 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714e3d1d_490f_40b4_ac26_0af71e16daa4.slice/crio-ff12ba3403aebf58308a357f0f112d7466fb7032a2c1717cae6e08d97474a4c6 WatchSource:0}: Error finding container ff12ba3403aebf58308a357f0f112d7466fb7032a2c1717cae6e08d97474a4c6: Status 404 returned error can't find the container with id ff12ba3403aebf58308a357f0f112d7466fb7032a2c1717cae6e08d97474a4c6 Nov 03 20:14:04.333025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.332944 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" event={"ID":"714e3d1d-490f-40b4-ac26-0af71e16daa4","Type":"ContainerStarted","Data":"ff12ba3403aebf58308a357f0f112d7466fb7032a2c1717cae6e08d97474a4c6"} Nov 03 20:14:04.334476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.334447 5815 generic.go:356] "Generic (PLEG): container finished" podID="d7f85b59-de69-4aed-b299-f35795ff644a" containerID="72441ff27637ca49e46f8033b8c4f8c95396275f695471701c042d0aeef64c1b" exitCode=0 Nov 03 20:14:04.334611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.334485 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" event={"ID":"d7f85b59-de69-4aed-b299-f35795ff644a","Type":"ContainerDied","Data":"72441ff27637ca49e46f8033b8c4f8c95396275f695471701c042d0aeef64c1b"} Nov 03 20:14:04.340753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.340731 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4"] Nov 03 20:14:04.344552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.344531 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.348353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.348335 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b13a14-openshift-default-dockercfg-r6zzp\"" Nov 03 20:14:04.361107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.361085 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4"] Nov 03 20:14:04.444425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444402 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444432 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444464 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444556 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444611 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444653 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7lb\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444690 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444720 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.444802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.444762 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545347 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545394 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545427 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545464 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545640 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545740 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.545799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545781 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7lb\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.546116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.545889 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.546116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.546024 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.546200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.546115 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.546296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.546265 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.546615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.546422 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.547988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.547964 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.548069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.548037 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.558313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.558292 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.558520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.558491 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7lb\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.657706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.657624 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:04.950497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.950472 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4"] Nov 03 20:14:04.960270 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:04.960244 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e23e11c_6c2f_4130_b713_d8615d6615b7.slice/crio-66c93ea1b96e267a17f64e0290a6c400a9876c4683f7179a72947f3b9fe88289 WatchSource:0}: Error finding container 66c93ea1b96e267a17f64e0290a6c400a9876c4683f7179a72947f3b9fe88289: Status 404 returned error can't find the container with id 66c93ea1b96e267a17f64e0290a6c400a9876c4683f7179a72947f3b9fe88289 Nov 03 20:14:04.962827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.962787 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:04.962898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.962855 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:04.962898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:04.962883 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:05.344948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.344920 5815 generic.go:356] "Generic (PLEG): container finished" podID="714e3d1d-490f-40b4-ac26-0af71e16daa4" containerID="0438d9a58f947b01063972b691cbc1118a010579874c8f3b639a0d047eedd678" exitCode=0 Nov 03 20:14:05.345143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.345008 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" event={"ID":"714e3d1d-490f-40b4-ac26-0af71e16daa4","Type":"ContainerDied","Data":"0438d9a58f947b01063972b691cbc1118a010579874c8f3b639a0d047eedd678"} Nov 03 20:14:05.347454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.346890 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" event={"ID":"1e23e11c-6c2f-4130-b713-d8615d6615b7","Type":"ContainerStarted","Data":"fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192"} Nov 03 20:14:05.347454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.346937 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" event={"ID":"1e23e11c-6c2f-4130-b713-d8615d6615b7","Type":"ContainerStarted","Data":"66c93ea1b96e267a17f64e0290a6c400a9876c4683f7179a72947f3b9fe88289"} Nov 03 20:14:05.386043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.385985 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podStartSLOduration=1.3859670880000001 podStartE2EDuration="1.385967088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:05.384984621 +0000 UTC m=+754.915330294" watchObservedRunningTime="2026-11-03 20:14:05.385967088 +0000 UTC m=+754.916312761" Nov 03 20:14:05.398876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.398848 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s"] Nov 03 20:14:05.402763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.402739 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.413893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.413869 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s"] Nov 03 20:14:05.454144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.454112 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-559pf\" (UniqueName: \"kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.454292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.454256 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.454292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.454284 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.507286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.507257 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:05.556102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.556062 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-559pf\" (UniqueName: \"kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.585765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.571721 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.586389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.585652 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.586561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.586353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.603696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.603596 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.604081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.604057 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-559pf\" (UniqueName: \"kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf\") pod \"maas-api-revoke-sub-ae855749-s5f9s\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.658632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.658597 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:05.693530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.692905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca\") pod \"d7f85b59-de69-4aed-b299-f35795ff644a\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " Nov 03 20:14:05.693530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.692979 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl\" (UniqueName: \"kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl\") pod \"d7f85b59-de69-4aed-b299-f35795ff644a\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " Nov 03 20:14:05.693530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.693078 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca\") pod \"d7f85b59-de69-4aed-b299-f35795ff644a\" (UID: \"d7f85b59-de69-4aed-b299-f35795ff644a\") " Nov 03 20:14:05.695612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.694317 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca" pod "d7f85b59-de69-4aed-b299-f35795ff644a" (UID: "d7f85b59-de69-4aed-b299-f35795ff644a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:05.699005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.698949 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca" pod "d7f85b59-de69-4aed-b299-f35795ff644a" (UID: "d7f85b59-de69-4aed-b299-f35795ff644a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:05.699005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.698969 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl" pod "d7f85b59-de69-4aed-b299-f35795ff644a" (UID: "d7f85b59-de69-4aed-b299-f35795ff644a"). InnerVolumeSpecName "kube-api-access-rbmzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:05.727140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.727112 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:05.796930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.796850 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:05.796930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.796896 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmzl\" (UniqueName: \"kubernetes.io/projected/d7f85b59-de69-4aed-b299-f35795ff644a-kube-api-access-rbmzl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:05.796930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:05.796913 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7f85b59-de69-4aed-b299-f35795ff644a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:06.034182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.034158 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s"] Nov 03 20:14:06.035916 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:06.035887 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6d7ce9_e29b_494f_9e1c_fbeb80e9cc4a.slice/crio-7e57066386480b7ae6ca3bffaadb3f5bfc548cb63b78b930f2539079fbb0bf1d WatchSource:0}: Error finding container 7e57066386480b7ae6ca3bffaadb3f5bfc548cb63b78b930f2539079fbb0bf1d: Status 404 returned error can't find the container with id 7e57066386480b7ae6ca3bffaadb3f5bfc548cb63b78b930f2539079fbb0bf1d Nov 03 20:14:06.377873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.363944 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" event={"ID":"d7f85b59-de69-4aed-b299-f35795ff644a","Type":"ContainerDied","Data":"4480a56718c85ea9501b0e258512101039c3cf504b7cb33ea8241ae9aa7f179c"} Nov 03 20:14:06.377873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.364005 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4480a56718c85ea9501b0e258512101039c3cf504b7cb33ea8241ae9aa7f179c" Nov 03 20:14:06.377873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.364084 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm" Nov 03 20:14:06.377873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.373747 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" event={"ID":"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a","Type":"ContainerStarted","Data":"7e57066386480b7ae6ca3bffaadb3f5bfc548cb63b78b930f2539079fbb0bf1d"} Nov 03 20:14:06.604764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.604740 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:06.610341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.610318 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca\") pod \"714e3d1d-490f-40b4-ac26-0af71e16daa4\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " Nov 03 20:14:06.610449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.610389 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca\") pod \"714e3d1d-490f-40b4-ac26-0af71e16daa4\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " Nov 03 20:14:06.610555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.610462 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8\" (UniqueName: \"kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8\") pod \"714e3d1d-490f-40b4-ac26-0af71e16daa4\" (UID: \"714e3d1d-490f-40b4-ac26-0af71e16daa4\") " Nov 03 20:14:06.610671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.610651 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca" pod "714e3d1d-490f-40b4-ac26-0af71e16daa4" (UID: "714e3d1d-490f-40b4-ac26-0af71e16daa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:06.610780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.610767 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:06.612639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.612614 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8" pod "714e3d1d-490f-40b4-ac26-0af71e16daa4" (UID: "714e3d1d-490f-40b4-ac26-0af71e16daa4"). InnerVolumeSpecName "kube-api-access-jn7q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:06.612746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.612722 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca" pod "714e3d1d-490f-40b4-ac26-0af71e16daa4" (UID: "714e3d1d-490f-40b4-ac26-0af71e16daa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:06.659149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.659122 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:14:06.659252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.659175 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:14:06.711802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.711772 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714e3d1d-490f-40b4-ac26-0af71e16daa4-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:06.711802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:06.711798 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7q8\" (UniqueName: \"kubernetes.io/projected/714e3d1d-490f-40b4-ac26-0af71e16daa4-kube-api-access-jn7q8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:07.404696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.404663 5815 generic.go:356] "Generic (PLEG): container finished" podID="2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" containerID="9d2bfdfa4e8dfc8ee54dd6e199055f5413267aa58f80e837c7c6f16eacc05dbe" exitCode=0 Nov 03 20:14:07.404892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.404750 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" event={"ID":"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a","Type":"ContainerDied","Data":"9d2bfdfa4e8dfc8ee54dd6e199055f5413267aa58f80e837c7c6f16eacc05dbe"} Nov 03 20:14:07.406146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.406125 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" Nov 03 20:14:07.406267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.406165 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8" event={"ID":"714e3d1d-490f-40b4-ac26-0af71e16daa4","Type":"ContainerDied","Data":"ff12ba3403aebf58308a357f0f112d7466fb7032a2c1717cae6e08d97474a4c6"} Nov 03 20:14:07.406267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.406202 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff12ba3403aebf58308a357f0f112d7466fb7032a2c1717cae6e08d97474a4c6" Nov 03 20:14:07.658886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.658858 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:14:07.659261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:07.658913 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:14:08.625643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.625615 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:08.658891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.658863 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:14:08.659271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.658912 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:14:08.733399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.733361 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca\") pod \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " Nov 03 20:14:08.733586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.733419 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca\") pod \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " Nov 03 20:14:08.733586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.733490 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf\" (UniqueName: \"kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf\") pod \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\" (UID: \"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a\") " Nov 03 20:14:08.733908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.733869 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca" pod "2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" (UID: "2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:08.735832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.735798 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca" pod "2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" (UID: "2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:08.735832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.735819 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf" pod "2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" (UID: "2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a"). InnerVolumeSpecName "kube-api-access-559pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:08.835317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.835286 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:08.835317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.835314 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:08.835317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:08.835325 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-559pf\" (UniqueName: \"kubernetes.io/projected/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a-kube-api-access-559pf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:09.419805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:09.419768 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" event={"ID":"2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a","Type":"ContainerDied","Data":"7e57066386480b7ae6ca3bffaadb3f5bfc548cb63b78b930f2539079fbb0bf1d"} Nov 03 20:14:09.420004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:09.419825 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e57066386480b7ae6ca3bffaadb3f5bfc548cb63b78b930f2539079fbb0bf1d" Nov 03 20:14:09.420004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:09.419891 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s" Nov 03 20:14:09.659050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:09.659011 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:14:09.659411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:09.659073 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:14:10.658652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:10.658617 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:14:10.658830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:10.658666 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 03 20:14:11.756120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.756087 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:11.756657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.756434 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:11.757644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.757621 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:14:11.887848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.887809 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p"] Nov 03 20:14:11.888307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888289 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d7f85b59-de69-4aed-b299-f35795ff644a" containerName="revoke-keys" Nov 03 20:14:11.888405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888309 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f85b59-de69-4aed-b299-f35795ff644a" containerName="revoke-keys" Nov 03 20:14:11.888405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888324 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" containerName="revoke-keys" Nov 03 20:14:11.888405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888331 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" containerName="revoke-keys" Nov 03 20:14:11.888405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888363 5815 cpu_manager.go:414] "containerMap: removing container" podUID="714e3d1d-490f-40b4-ac26-0af71e16daa4" containerName="revoke-keys" Nov 03 20:14:11.888405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888370 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714e3d1d-490f-40b4-ac26-0af71e16daa4" containerName="revoke-keys" Nov 03 20:14:11.888664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888480 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="714e3d1d-490f-40b4-ac26-0af71e16daa4" containerName="revoke-keys" Nov 03 20:14:11.888664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888499 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f85b59-de69-4aed-b299-f35795ff644a" containerName="revoke-keys" Nov 03 20:14:11.888664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.888530 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" containerName="revoke-keys" Nov 03 20:14:11.891997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.891978 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:11.900259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.900226 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:11.900985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.900616 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:14:11.905750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.905731 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p"] Nov 03 20:14:11.967613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.967581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:11.967780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.967757 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lhf\" (UniqueName: \"kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:11.967858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:11.967792 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.069037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.068947 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lhf\" (UniqueName: \"kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.069037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.068989 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.069272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.069038 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.069778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.069753 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.071836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.071808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.081432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.081399 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lhf\" (UniqueName: \"kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf\") pod \"maas-api-revoke-sub-cd027bf1-z984p\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.206905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.206877 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:12.491147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:12.491082 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p"] Nov 03 20:14:12.498851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:12.498819 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b692a65_f0f3_424a_bae6_e8ed8e3df112.slice/crio-286c4ec8f8132dcdd3a81f5c546fedc89b03a62265f8e8179d479d484e5d7e01 WatchSource:0}: Error finding container 286c4ec8f8132dcdd3a81f5c546fedc89b03a62265f8e8179d479d484e5d7e01: Status 404 returned error can't find the container with id 286c4ec8f8132dcdd3a81f5c546fedc89b03a62265f8e8179d479d484e5d7e01 Nov 03 20:14:13.457347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:13.457317 5815 generic.go:356] "Generic (PLEG): container finished" podID="2b692a65-f0f3-424a-bae6-e8ed8e3df112" containerID="58cd479b94ecb3da8fc80c9321217e0a2f6bcdc5d4ec72a8ff2f65da485238e4" exitCode=0 Nov 03 20:14:13.457744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:13.457463 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" event={"ID":"2b692a65-f0f3-424a-bae6-e8ed8e3df112","Type":"ContainerDied","Data":"58cd479b94ecb3da8fc80c9321217e0a2f6bcdc5d4ec72a8ff2f65da485238e4"} Nov 03 20:14:13.457744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:13.457542 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" event={"ID":"2b692a65-f0f3-424a-bae6-e8ed8e3df112","Type":"ContainerStarted","Data":"286c4ec8f8132dcdd3a81f5c546fedc89b03a62265f8e8179d479d484e5d7e01"} Nov 03 20:14:13.869312 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:14:13.869269 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:14:14.604285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.604263 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:14.694582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.694554 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca\") pod \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " Nov 03 20:14:14.694757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.694605 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca\") pod \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " Nov 03 20:14:14.694757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.694670 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf\" (UniqueName: \"kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf\") pod \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\" (UID: \"2b692a65-f0f3-424a-bae6-e8ed8e3df112\") " Nov 03 20:14:14.695119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.695010 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca" pod "2b692a65-f0f3-424a-bae6-e8ed8e3df112" (UID: "2b692a65-f0f3-424a-bae6-e8ed8e3df112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:14.697008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.696986 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf" pod "2b692a65-f0f3-424a-bae6-e8ed8e3df112" (UID: "2b692a65-f0f3-424a-bae6-e8ed8e3df112"). InnerVolumeSpecName "kube-api-access-45lhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:14.697244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.697221 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca" pod "2b692a65-f0f3-424a-bae6-e8ed8e3df112" (UID: "2b692a65-f0f3-424a-bae6-e8ed8e3df112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:14.796452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.796375 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:14.796452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.796412 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b692a65-f0f3-424a-bae6-e8ed8e3df112-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:14.796452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:14.796429 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lhf\" (UniqueName: \"kubernetes.io/projected/2b692a65-f0f3-424a-bae6-e8ed8e3df112-kube-api-access-45lhf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:15.505571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:15.503571 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" event={"ID":"2b692a65-f0f3-424a-bae6-e8ed8e3df112","Type":"ContainerDied","Data":"286c4ec8f8132dcdd3a81f5c546fedc89b03a62265f8e8179d479d484e5d7e01"} Nov 03 20:14:15.505571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:15.503638 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="286c4ec8f8132dcdd3a81f5c546fedc89b03a62265f8e8179d479d484e5d7e01" Nov 03 20:14:15.505571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:15.503703 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p" Nov 03 20:14:16.362763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.362729 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq"] Nov 03 20:14:16.363387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.363372 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2b692a65-f0f3-424a-bae6-e8ed8e3df112" containerName="revoke-keys" Nov 03 20:14:16.363435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.363390 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b692a65-f0f3-424a-bae6-e8ed8e3df112" containerName="revoke-keys" Nov 03 20:14:16.363549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.363536 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b692a65-f0f3-424a-bae6-e8ed8e3df112" containerName="revoke-keys" Nov 03 20:14:16.367232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.367213 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.371612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.371582 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:14:16.371750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.371649 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:16.373343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.373317 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq"] Nov 03 20:14:16.418011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.417978 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.418153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.418089 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.418203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.418156 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpsh\" (UniqueName: \"kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.519241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.519206 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.519386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.519266 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.519386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.519335 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpsh\" (UniqueName: \"kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.519974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.519939 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.521730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.521703 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.534223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.534200 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpsh\" (UniqueName: \"kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh\") pod \"maas-api-revoke-sub-a28c5d52-x8fzq\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:16.682700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:16.682615 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:17.004226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:17.004197 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq"] Nov 03 20:14:17.006835 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:17.006663 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900dc148_7955_4626_af6d_3a0ada8f6298.slice/crio-16be38d1592bea7eadb88d3ffc8a5e6dca96973f2ddd31f8cd178e2a919a857b WatchSource:0}: Error finding container 16be38d1592bea7eadb88d3ffc8a5e6dca96973f2ddd31f8cd178e2a919a857b: Status 404 returned error can't find the container with id 16be38d1592bea7eadb88d3ffc8a5e6dca96973f2ddd31f8cd178e2a919a857b Nov 03 20:14:17.525930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:17.525897 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" event={"ID":"900dc148-7955-4626-af6d-3a0ada8f6298","Type":"ContainerStarted","Data":"f9beb736435fb4d6074b28cd9f27584b3dba496cc9c6fd23261eb76a4779d7bf"} Nov 03 20:14:17.526318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:17.525958 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" event={"ID":"900dc148-7955-4626-af6d-3a0ada8f6298","Type":"ContainerStarted","Data":"16be38d1592bea7eadb88d3ffc8a5e6dca96973f2ddd31f8cd178e2a919a857b"} Nov 03 20:14:17.546179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:17.546104 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" podStartSLOduration=1.546092859 podStartE2EDuration="1.546092859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:17.543735326 +0000 UTC m=+767.074081000" watchObservedRunningTime="2026-11-03 20:14:17.546092859 +0000 UTC m=+767.076438566" Nov 03 20:14:18.535837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:18.535793 5815 generic.go:356] "Generic (PLEG): container finished" podID="900dc148-7955-4626-af6d-3a0ada8f6298" containerID="f9beb736435fb4d6074b28cd9f27584b3dba496cc9c6fd23261eb76a4779d7bf" exitCode=0 Nov 03 20:14:18.536500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:18.535965 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" event={"ID":"900dc148-7955-4626-af6d-3a0ada8f6298","Type":"ContainerDied","Data":"f9beb736435fb4d6074b28cd9f27584b3dba496cc9c6fd23261eb76a4779d7bf"} Nov 03 20:14:20.065373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.065065 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:20.103693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.102911 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca\") pod \"900dc148-7955-4626-af6d-3a0ada8f6298\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " Nov 03 20:14:20.103693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.102991 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh\" (UniqueName: \"kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh\") pod \"900dc148-7955-4626-af6d-3a0ada8f6298\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " Nov 03 20:14:20.103693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.103098 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca\") pod \"900dc148-7955-4626-af6d-3a0ada8f6298\" (UID: \"900dc148-7955-4626-af6d-3a0ada8f6298\") " Nov 03 20:14:20.114528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.104154 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca" pod "900dc148-7955-4626-af6d-3a0ada8f6298" (UID: "900dc148-7955-4626-af6d-3a0ada8f6298"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:20.167532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.166653 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh" pod "900dc148-7955-4626-af6d-3a0ada8f6298" (UID: "900dc148-7955-4626-af6d-3a0ada8f6298"). InnerVolumeSpecName "kube-api-access-thpsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:20.193536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.179613 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca" pod "900dc148-7955-4626-af6d-3a0ada8f6298" (UID: "900dc148-7955-4626-af6d-3a0ada8f6298"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:20.224537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.204152 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:20.224817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.224795 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thpsh\" (UniqueName: \"kubernetes.io/projected/900dc148-7955-4626-af6d-3a0ada8f6298-kube-api-access-thpsh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:20.231597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.231568 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900dc148-7955-4626-af6d-3a0ada8f6298-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:20.559769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.559737 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" Nov 03 20:14:20.559769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.559744 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq" event={"ID":"900dc148-7955-4626-af6d-3a0ada8f6298","Type":"ContainerDied","Data":"16be38d1592bea7eadb88d3ffc8a5e6dca96973f2ddd31f8cd178e2a919a857b"} Nov 03 20:14:20.560000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:20.559804 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16be38d1592bea7eadb88d3ffc8a5e6dca96973f2ddd31f8cd178e2a919a857b" Nov 03 20:14:26.573664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.573627 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v"] Nov 03 20:14:26.574317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.574294 5815 cpu_manager.go:414] "containerMap: removing container" podUID="900dc148-7955-4626-af6d-3a0ada8f6298" containerName="revoke-keys" Nov 03 20:14:26.574317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.574314 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900dc148-7955-4626-af6d-3a0ada8f6298" containerName="revoke-keys" Nov 03 20:14:26.574554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.574419 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="900dc148-7955-4626-af6d-3a0ada8f6298" containerName="revoke-keys" Nov 03 20:14:26.590465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.588909 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.597265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.596322 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:26.597265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.596434 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:14:26.615536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.611706 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtw2v\" (UniqueName: \"kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.615536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.611764 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.615536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.611849 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.640850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.640821 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v"] Nov 03 20:14:26.712534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.712475 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtw2v\" (UniqueName: \"kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.712703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.712554 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.712703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.712610 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.713237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.713213 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.715094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.715064 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.727375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.727354 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtw2v\" (UniqueName: \"kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v\") pod \"maas-api-revoke-sub-c2c0b564-fg75v\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:26.787314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.787282 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x"] Nov 03 20:14:26.793866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.793821 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x"] Nov 03 20:14:26.794007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.793947 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.799492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.799453 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"e2e-conflict-a-256c5a80-kserve-self-signed-certs\"" Nov 03 20:14:26.928584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.920228 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.928584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.920323 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.928584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.920365 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.928584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.920419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnbk9\" (UniqueName: \"kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.928584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.920471 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.928584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.920538 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:26.940958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:26.940924 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021096 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021183 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021222 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021297 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnbk9\" (UniqueName: \"kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021358 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021405 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.021752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.021560 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.022240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.022049 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.022240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.022195 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.024442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.024419 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.024860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.024844 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.040485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.036111 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnbk9\" (UniqueName: \"kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9\") pod \"e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.112072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.112038 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:27.268621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.268591 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v"] Nov 03 20:14:27.398978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.398952 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x"] Nov 03 20:14:27.399645 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:27.399611 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0ca2df6_5587_4d4e_b59e_48b9f7d326dd.slice/crio-cffdc54a41cc67301d6a57e3ad2d7f9106e96a5cd75b8e748f00bb56468b8ece WatchSource:0}: Error finding container cffdc54a41cc67301d6a57e3ad2d7f9106e96a5cd75b8e748f00bb56468b8ece: Status 404 returned error can't find the container with id cffdc54a41cc67301d6a57e3ad2d7f9106e96a5cd75b8e748f00bb56468b8ece Nov 03 20:14:27.650048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.650012 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" event={"ID":"65e6ad83-0421-4b0e-9ab8-654eb386beb0","Type":"ContainerStarted","Data":"cdef2c9835372a8f0c4066a10601634d3ada52334abb59176aa28c5f4f7580d1"} Nov 03 20:14:27.650465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.650074 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" event={"ID":"65e6ad83-0421-4b0e-9ab8-654eb386beb0","Type":"ContainerStarted","Data":"6cf7eabf1d3cd868c6f2b5662946feb8532833557e7e3c0333a7f25b25609281"} Nov 03 20:14:27.651681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.651654 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" event={"ID":"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd","Type":"ContainerStarted","Data":"b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5"} Nov 03 20:14:27.651818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.651694 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" event={"ID":"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd","Type":"ContainerStarted","Data":"cffdc54a41cc67301d6a57e3ad2d7f9106e96a5cd75b8e748f00bb56468b8ece"} Nov 03 20:14:27.670945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.670895 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" podStartSLOduration=1.670880274 podStartE2EDuration="1.670880274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.668812143 +0000 UTC m=+777.199157819" watchObservedRunningTime="2026-11-03 20:14:27.670880274 +0000 UTC m=+777.201226369" Nov 03 20:14:27.689713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:27.689673 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" podStartSLOduration=1.689660218 podStartE2EDuration="1.689660218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.687910167 +0000 UTC m=+777.218255840" watchObservedRunningTime="2026-11-03 20:14:27.689660218 +0000 UTC m=+777.220005890" Nov 03 20:14:28.112225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:28.112192 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:28.357040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:28.356988 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:14:28.659200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:28.659166 5815 generic.go:356] "Generic (PLEG): container finished" podID="65e6ad83-0421-4b0e-9ab8-654eb386beb0" containerID="cdef2c9835372a8f0c4066a10601634d3ada52334abb59176aa28c5f4f7580d1" exitCode=0 Nov 03 20:14:28.659674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:28.659243 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" event={"ID":"65e6ad83-0421-4b0e-9ab8-654eb386beb0","Type":"ContainerDied","Data":"cdef2c9835372a8f0c4066a10601634d3ada52334abb59176aa28c5f4f7580d1"} Nov 03 20:14:29.901477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.898995 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:29.970534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.969902 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v\" (UniqueName: \"kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v\") pod \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " Nov 03 20:14:29.970534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.970000 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca\") pod \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " Nov 03 20:14:29.970534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.970077 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca\") pod \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\" (UID: \"65e6ad83-0421-4b0e-9ab8-654eb386beb0\") " Nov 03 20:14:29.971404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.971080 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca" pod "65e6ad83-0421-4b0e-9ab8-654eb386beb0" (UID: "65e6ad83-0421-4b0e-9ab8-654eb386beb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:29.973852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.973823 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v" pod "65e6ad83-0421-4b0e-9ab8-654eb386beb0" (UID: "65e6ad83-0421-4b0e-9ab8-654eb386beb0"). InnerVolumeSpecName "kube-api-access-xtw2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:29.976798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:29.976769 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca" pod "65e6ad83-0421-4b0e-9ab8-654eb386beb0" (UID: "65e6ad83-0421-4b0e-9ab8-654eb386beb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:30.072190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:30.072021 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtw2v\" (UniqueName: \"kubernetes.io/projected/65e6ad83-0421-4b0e-9ab8-654eb386beb0-kube-api-access-xtw2v\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:30.072190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:30.072058 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:30.072190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:30.072074 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e6ad83-0421-4b0e-9ab8-654eb386beb0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:30.783752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:30.783715 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" event={"ID":"65e6ad83-0421-4b0e-9ab8-654eb386beb0","Type":"ContainerDied","Data":"6cf7eabf1d3cd868c6f2b5662946feb8532833557e7e3c0333a7f25b25609281"} Nov 03 20:14:30.783980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:30.783967 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf7eabf1d3cd868c6f2b5662946feb8532833557e7e3c0333a7f25b25609281" Nov 03 20:14:30.784134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:30.784124 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v" Nov 03 20:14:32.267675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:32.267639 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:32.268413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:32.268379 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.113533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:37.113359 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:37.137539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:37.137474 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:37.900539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:37.900475 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:14:39.478531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:39.477603 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:39.478531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:39.477713 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.502530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:39.493622 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:39.502530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:39.493691 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.279798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.279754 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx"] Nov 03 20:14:43.280370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.280353 5815 cpu_manager.go:414] "containerMap: removing container" podUID="65e6ad83-0421-4b0e-9ab8-654eb386beb0" containerName="revoke-keys" Nov 03 20:14:43.280439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.280371 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e6ad83-0421-4b0e-9ab8-654eb386beb0" containerName="revoke-keys" Nov 03 20:14:43.280547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.280534 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e6ad83-0421-4b0e-9ab8-654eb386beb0" containerName="revoke-keys" Nov 03 20:14:43.284201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.284176 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.289289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.289270 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-466f83-openshift-default-dockercfg-6pzmx\"" Nov 03 20:14:43.298061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.297770 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx"] Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.318965 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7ct6\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.318998 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319041 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319061 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319085 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319125 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319148 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319183 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.319540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.319204 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420080 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420133 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420152 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420184 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420218 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420300 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420333 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420558 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ct6\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.420764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420729 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.421020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.420995 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.421116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.421061 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.421116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.421099 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.421116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.421103 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.423076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.423054 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.423294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.423278 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.432806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.432780 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.432980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.432966 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ct6\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.606911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.606824 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:43.850774 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:14:43.850741 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:14:43.901144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.901060 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx"] Nov 03 20:14:43.906091 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:43.906059 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805307b8_5902_4bc3_8d13_002a1cc438c6.slice/crio-0cee4ecf61eed5a6d2ff5540a99f28e0f235f55c5cd4b71e240b57ba1649ecc5 WatchSource:0}: Error finding container 0cee4ecf61eed5a6d2ff5540a99f28e0f235f55c5cd4b71e240b57ba1649ecc5: Status 404 returned error can't find the container with id 0cee4ecf61eed5a6d2ff5540a99f28e0f235f55c5cd4b71e240b57ba1649ecc5 Nov 03 20:14:43.909142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.909099 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:43.909256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.909190 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:43.909256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:43.909231 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:44.021546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.020310 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" event={"ID":"805307b8-5902-4bc3-8d13-002a1cc438c6","Type":"ContainerStarted","Data":"1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4"} Nov 03 20:14:44.021546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.020380 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" event={"ID":"805307b8-5902-4bc3-8d13-002a1cc438c6","Type":"ContainerStarted","Data":"0cee4ecf61eed5a6d2ff5540a99f28e0f235f55c5cd4b71e240b57ba1649ecc5"} Nov 03 20:14:44.046500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.046452 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podStartSLOduration=1.046439207 podStartE2EDuration="1.046439207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:44.043814013 +0000 UTC m=+793.574159688" watchObservedRunningTime="2026-11-03 20:14:44.046439207 +0000 UTC m=+793.576784878" Nov 03 20:14:44.151581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.139827 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:14:44.422790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.422721 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9"] Nov 03 20:14:44.426929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.426911 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.436540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.436070 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3631f-openshift-default-dockercfg-2vnzz\"" Nov 03 20:14:44.442849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.442830 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9"] Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533107 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533159 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533191 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533230 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533299 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533366 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533397 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533437 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdxz\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.537530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.533477 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.607897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.607873 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:44.609549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.609495 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:44.609689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.609584 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:44.634578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634552 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdxz\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.634717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634586 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.634717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634618 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.634717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634633 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.634717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634662 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.634717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634690 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.635007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634749 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.635007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634814 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.635007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.634836 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.635461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.635439 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.635900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.635882 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.636076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.636063 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.636438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.636421 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.637332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.637306 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.639658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.639634 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.641207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.641185 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.648676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.648633 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.649896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.649863 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdxz\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:44.752895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:44.752858 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:45.136412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:45.136380 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9"] Nov 03 20:14:45.137890 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:45.137860 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a20ebd_87e0_4fa7_82cb_8b5041895eaa.slice/crio-4a9807bf5b485d59af74aa1f79ce693fed27d1c924dbb7a570fd58182e75e0f2 WatchSource:0}: Error finding container 4a9807bf5b485d59af74aa1f79ce693fed27d1c924dbb7a570fd58182e75e0f2: Status 404 returned error can't find the container with id 4a9807bf5b485d59af74aa1f79ce693fed27d1c924dbb7a570fd58182e75e0f2 Nov 03 20:14:45.140244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:45.140212 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:45.140358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:45.140290 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:45.140424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:45.140360 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:14:45.608217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:45.608183 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:45.608857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:45.608809 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:46.130060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.129706 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" event={"ID":"20a20ebd-87e0-4fa7-82cb-8b5041895eaa","Type":"ContainerStarted","Data":"9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1"} Nov 03 20:14:46.130060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.129766 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" event={"ID":"20a20ebd-87e0-4fa7-82cb-8b5041895eaa","Type":"ContainerStarted","Data":"4a9807bf5b485d59af74aa1f79ce693fed27d1c924dbb7a570fd58182e75e0f2"} Nov 03 20:14:46.155615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.155568 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podStartSLOduration=2.155550984 podStartE2EDuration="2.155550984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:46.153054378 +0000 UTC m=+795.683400050" watchObservedRunningTime="2026-11-03 20:14:46.155550984 +0000 UTC m=+795.685896669" Nov 03 20:14:46.607551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.607488 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:46.607816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.607790 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:46.753547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.753226 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:46.754853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.754602 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:46.754853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:46.754645 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:47.611527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:47.610864 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:47.611527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:47.610922 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:47.754248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:47.754097 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:47.754248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:47.754169 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:48.611533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:48.609868 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:48.611533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:48.609924 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:48.753723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:48.753693 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:48.753954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:48.753935 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:49.610533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:49.609884 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:49.610533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:49.609954 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:49.754009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:49.753969 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:49.754203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:49.754025 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:50.615767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:50.615730 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:14:50.616391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:50.616357 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 20:14:50.753830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:50.753779 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:50.754044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:50.753848 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:51.753677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:51.753643 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 20:14:51.754185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:51.753693 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 20:14:52.309727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:52.309696 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:52.310498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:52.310476 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:52.311582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:52.311564 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:14:52.832618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:52.832588 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:52.833294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:52.833276 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:52.835703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:52.835680 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:14:59.092821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.092784 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn"] Nov 03 20:14:59.099541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.099497 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.112264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.112234 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn"] Nov 03 20:14:59.113773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.113748 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"distinct-2-w4-466f83-kserve-self-signed-certs\"" Nov 03 20:14:59.221154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.221119 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.221364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.221171 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bwp\" (UniqueName: \"kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.221364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.221207 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.221364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.221306 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.221364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.221344 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.221626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.221546 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.329532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.326119 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.329532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.326188 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.329532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.326223 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bwp\" (UniqueName: \"kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.329532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.326248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.329532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.326282 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.329532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.326315 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.330922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.330896 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.332472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.332452 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.332739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.332664 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.343218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.343153 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.355411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.346753 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bwp\" (UniqueName: \"kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.363677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.363113 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs\") pod \"distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.420924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.420897 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:14:59.710137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:14:59.710116 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn"] Nov 03 20:14:59.713580 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:14:59.713545 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7781320_da95_4c12_9645_b22679a15067.slice/crio-ed1ca26004bd4d68e0a41a918806d12cf55fdbfcbff05fe33af835de41b1625f WatchSource:0}: Error finding container ed1ca26004bd4d68e0a41a918806d12cf55fdbfcbff05fe33af835de41b1625f: Status 404 returned error can't find the container with id ed1ca26004bd4d68e0a41a918806d12cf55fdbfcbff05fe33af835de41b1625f Nov 03 20:15:00.152388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.152294 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl"] Nov 03 20:15:00.156238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.156216 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj"] Nov 03 20:15:00.156373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.156334 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.160274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.160254 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d"] Nov 03 20:15:00.160405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.160348 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.164443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.164350 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.167818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.164870 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:15:00.170429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.170375 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj"] Nov 03 20:15:00.178239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.178221 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl"] Nov 03 20:15:00.188085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.188057 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d"] Nov 03 20:15:00.195879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.193675 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m"] Nov 03 20:15:00.200524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.198127 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.201551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.201131 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m"] Nov 03 20:15:00.237909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.237877 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbt7c\" (UniqueName: \"kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c\") pod \"maas-api-key-cleanup-29895615-x8dnj\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.238052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.237954 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.238052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.238028 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkmg\" (UniqueName: \"kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.238177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.238065 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-x8dnj\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.238177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.238092 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.238177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.238150 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.238335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.238191 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7qv8\" (UniqueName: \"kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.238335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.238234 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67f2l\" (UniqueName: \"kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.265710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.265684 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" event={"ID":"d7781320-da95-4c12-9645-b22679a15067","Type":"ContainerStarted","Data":"3caa77faedd47ce304cf81db691e43247416c42a6dc3d25cb6daf60ede9d251c"} Nov 03 20:15:00.265828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.265719 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" event={"ID":"d7781320-da95-4c12-9645-b22679a15067","Type":"ContainerStarted","Data":"ed1ca26004bd4d68e0a41a918806d12cf55fdbfcbff05fe33af835de41b1625f"} Nov 03 20:15:00.288057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.288011 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podStartSLOduration=1.287994817 podStartE2EDuration="1.287994817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:00.284280718 +0000 UTC m=+809.814626391" watchObservedRunningTime="2026-11-03 20:15:00.287994817 +0000 UTC m=+809.818340491" Nov 03 20:15:00.339262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339235 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-x8dnj\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.339394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339266 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.339394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339306 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.339394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339340 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7qv8\" (UniqueName: \"kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.339650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339629 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67f2l\" (UniqueName: \"kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.339763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339746 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbt7c\" (UniqueName: \"kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c\") pod \"maas-api-key-cleanup-29895615-x8dnj\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.339834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.339907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.339890 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkmg\" (UniqueName: \"kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.340105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.340084 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.340181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.340138 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.340271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.340091 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895615-x8dnj\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.340783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.340765 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.352274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.352253 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7qv8\" (UniqueName: \"kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.352392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.352284 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbt7c\" (UniqueName: \"kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c\") pod \"maas-api-key-cleanup-29895615-x8dnj\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.352614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.352588 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67f2l\" (UniqueName: \"kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.355536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.353616 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkmg\" (UniqueName: \"kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.422043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.421970 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:00.479357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.479320 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:00.499599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.499573 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:00.502893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.502739 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:00.517753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.517723 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:00.780861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.780828 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl"] Nov 03 20:15:00.781185 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:00.781164 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2d3584_09d0_4549_9735_dcafcf867c0e.slice/crio-13530643366d94a463c00b01c2cf576c37c9c9756aff1d4058c9cfcda0dbf048 WatchSource:0}: Error finding container 13530643366d94a463c00b01c2cf576c37c9c9756aff1d4058c9cfcda0dbf048: Status 404 returned error can't find the container with id 13530643366d94a463c00b01c2cf576c37c9c9756aff1d4058c9cfcda0dbf048 Nov 03 20:15:00.822748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.822720 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj"] Nov 03 20:15:00.823788 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:00.823764 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c5cf8e_2afb_4ed6_9944_978c3cd59d6a.slice/crio-6dfc9711fc4585e8045571c29e39239b934b6145923d2e394d79df0f1604ca60 WatchSource:0}: Error finding container 6dfc9711fc4585e8045571c29e39239b934b6145923d2e394d79df0f1604ca60: Status 404 returned error can't find the container with id 6dfc9711fc4585e8045571c29e39239b934b6145923d2e394d79df0f1604ca60 Nov 03 20:15:00.863165 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:00.863104 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5079d318_0304_43b6_bf50_d517c1b9b217.slice/crio-82a584ecbc2ec9f12b654a060a678b49a499a14d944772d35a4539cfcaf3b2b7 WatchSource:0}: Error finding container 82a584ecbc2ec9f12b654a060a678b49a499a14d944772d35a4539cfcaf3b2b7: Status 404 returned error can't find the container with id 82a584ecbc2ec9f12b654a060a678b49a499a14d944772d35a4539cfcaf3b2b7 Nov 03 20:15:00.864356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.863167 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m"] Nov 03 20:15:00.867475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:00.867452 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d"] Nov 03 20:15:01.276230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:01.276198 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" event={"ID":"5079d318-0304-43b6-bf50-d517c1b9b217","Type":"ContainerStarted","Data":"82a584ecbc2ec9f12b654a060a678b49a499a14d944772d35a4539cfcaf3b2b7"} Nov 03 20:15:01.278386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:01.278323 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerStarted","Data":"080505df66089462f257ef5758c4983b21ee14da33745c321ff04eceeeea48ae"} Nov 03 20:15:01.280353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:01.280278 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" event={"ID":"4e2d3584-09d0-4549-9735-dcafcf867c0e","Type":"ContainerStarted","Data":"13530643366d94a463c00b01c2cf576c37c9c9756aff1d4058c9cfcda0dbf048"} Nov 03 20:15:01.282219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:01.282195 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerStarted","Data":"2061d42341b72c826d5f59568d4f10611372c8dc02cd300d79ab608777993c3e"} Nov 03 20:15:01.282303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:01.282247 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerStarted","Data":"6dfc9711fc4585e8045571c29e39239b934b6145923d2e394d79df0f1604ca60"} Nov 03 20:15:01.304258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:01.304220 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" podStartSLOduration=1.304208214 podStartE2EDuration="1.304208214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:01.303173222 +0000 UTC m=+810.833518894" watchObservedRunningTime="2026-11-03 20:15:01.304208214 +0000 UTC m=+810.834553885" Nov 03 20:15:02.292579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.292542 5815 generic.go:356] "Generic (PLEG): container finished" podID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerID="c7d23522cc62600a25d26d06d6d7e5ffb1c0b2035c0190b5015c412c2e63ea1a" exitCode=60 Nov 03 20:15:02.293053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.292609 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" event={"ID":"4e2d3584-09d0-4549-9735-dcafcf867c0e","Type":"ContainerDied","Data":"c7d23522cc62600a25d26d06d6d7e5ffb1c0b2035c0190b5015c412c2e63ea1a"} Nov 03 20:15:02.293053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.292863 5815 scope.go:122] "RemoveContainer" containerID="c7d23522cc62600a25d26d06d6d7e5ffb1c0b2035c0190b5015c412c2e63ea1a" Nov 03 20:15:02.294703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.294681 5815 generic.go:356] "Generic (PLEG): container finished" podID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerID="2061d42341b72c826d5f59568d4f10611372c8dc02cd300d79ab608777993c3e" exitCode=60 Nov 03 20:15:02.294805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.294721 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerDied","Data":"2061d42341b72c826d5f59568d4f10611372c8dc02cd300d79ab608777993c3e"} Nov 03 20:15:02.295017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.294991 5815 scope.go:122] "RemoveContainer" containerID="2061d42341b72c826d5f59568d4f10611372c8dc02cd300d79ab608777993c3e" Nov 03 20:15:02.297406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.296821 5815 generic.go:356] "Generic (PLEG): container finished" podID="5079d318-0304-43b6-bf50-d517c1b9b217" containerID="79eb6f2bada7821aba156271af79ac48d89e8f62b845ee6a5c5227808f8c7365" exitCode=60 Nov 03 20:15:02.297406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.296900 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" event={"ID":"5079d318-0304-43b6-bf50-d517c1b9b217","Type":"ContainerDied","Data":"79eb6f2bada7821aba156271af79ac48d89e8f62b845ee6a5c5227808f8c7365"} Nov 03 20:15:02.297406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.297220 5815 scope.go:122] "RemoveContainer" containerID="79eb6f2bada7821aba156271af79ac48d89e8f62b845ee6a5c5227808f8c7365" Nov 03 20:15:02.298781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.298760 5815 generic.go:356] "Generic (PLEG): container finished" podID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerID="ffae04c02238e1bab3f01af8fc3eaaa2cc4abb07afe9d82494d00573ef69e6a1" exitCode=60 Nov 03 20:15:02.298877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.298830 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerDied","Data":"ffae04c02238e1bab3f01af8fc3eaaa2cc4abb07afe9d82494d00573ef69e6a1"} Nov 03 20:15:02.299061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:02.299046 5815 scope.go:122] "RemoveContainer" containerID="ffae04c02238e1bab3f01af8fc3eaaa2cc4abb07afe9d82494d00573ef69e6a1" Nov 03 20:15:03.309164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.309137 5815 generic.go:356] "Generic (PLEG): container finished" podID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerID="4a9872f67d204509acb3a7642ec5be6f4202bfa5ef2586ebc9dfe1daff6fa4a4" exitCode=60 Nov 03 20:15:03.309601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.309225 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" event={"ID":"4e2d3584-09d0-4549-9735-dcafcf867c0e","Type":"ContainerDied","Data":"4a9872f67d204509acb3a7642ec5be6f4202bfa5ef2586ebc9dfe1daff6fa4a4"} Nov 03 20:15:03.309601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.309325 5815 scope.go:122] "RemoveContainer" containerID="c7d23522cc62600a25d26d06d6d7e5ffb1c0b2035c0190b5015c412c2e63ea1a" Nov 03 20:15:03.309601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.309557 5815 scope.go:122] "RemoveContainer" containerID="4a9872f67d204509acb3a7642ec5be6f4202bfa5ef2586ebc9dfe1daff6fa4a4" Nov 03 20:15:03.309813 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:03.309794 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl_odh-ai-gateway-infra(4e2d3584-09d0-4549-9735-dcafcf867c0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" Nov 03 20:15:03.311873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.311851 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerStarted","Data":"4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681"} Nov 03 20:15:03.314201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.314178 5815 generic.go:356] "Generic (PLEG): container finished" podID="5079d318-0304-43b6-bf50-d517c1b9b217" containerID="4c41a40810fb571c03e92a522281140984b6349a5d610454efac9cce947ce173" exitCode=60 Nov 03 20:15:03.314313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.314247 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" event={"ID":"5079d318-0304-43b6-bf50-d517c1b9b217","Type":"ContainerDied","Data":"4c41a40810fb571c03e92a522281140984b6349a5d610454efac9cce947ce173"} Nov 03 20:15:03.314533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.314490 5815 scope.go:122] "RemoveContainer" containerID="4c41a40810fb571c03e92a522281140984b6349a5d610454efac9cce947ce173" Nov 03 20:15:03.314874 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:03.314850 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m_odh-ai-gateway-infra(5079d318-0304-43b6-bf50-d517c1b9b217)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" Nov 03 20:15:03.316319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.316299 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerStarted","Data":"e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c"} Nov 03 20:15:03.347028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.347013 5815 scope.go:122] "RemoveContainer" containerID="79eb6f2bada7821aba156271af79ac48d89e8f62b845ee6a5c5227808f8c7365" Nov 03 20:15:03.364893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:03.364851 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" podStartSLOduration=3.36483586 podStartE2EDuration="3.36483586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:03.362621239 +0000 UTC m=+812.892966941" watchObservedRunningTime="2026-11-03 20:15:03.36483586 +0000 UTC m=+812.895181532" Nov 03 20:15:04.326008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.325978 5815 scope.go:122] "RemoveContainer" containerID="4a9872f67d204509acb3a7642ec5be6f4202bfa5ef2586ebc9dfe1daff6fa4a4" Nov 03 20:15:04.326537 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:04.326243 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl_odh-ai-gateway-infra(4e2d3584-09d0-4549-9735-dcafcf867c0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" Nov 03 20:15:04.328190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.328162 5815 generic.go:356] "Generic (PLEG): container finished" podID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerID="4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681" exitCode=60 Nov 03 20:15:04.328316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.328259 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerDied","Data":"4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681"} Nov 03 20:15:04.328380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.328325 5815 scope.go:122] "RemoveContainer" containerID="2061d42341b72c826d5f59568d4f10611372c8dc02cd300d79ab608777993c3e" Nov 03 20:15:04.328484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.328468 5815 scope.go:122] "RemoveContainer" containerID="4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681" Nov 03 20:15:04.328771 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:04.328749 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-x8dnj_odh-ai-gateway-infra(99c5cf8e-2afb-4ed6-9944-978c3cd59d6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" Nov 03 20:15:04.330315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.330298 5815 scope.go:122] "RemoveContainer" containerID="4c41a40810fb571c03e92a522281140984b6349a5d610454efac9cce947ce173" Nov 03 20:15:04.330530 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:04.330486 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m_odh-ai-gateway-infra(5079d318-0304-43b6-bf50-d517c1b9b217)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" Nov 03 20:15:04.332195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.332175 5815 generic.go:356] "Generic (PLEG): container finished" podID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerID="e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c" exitCode=60 Nov 03 20:15:04.332461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.332216 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerDied","Data":"e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c"} Nov 03 20:15:04.332461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.332401 5815 scope.go:122] "RemoveContainer" containerID="e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c" Nov 03 20:15:04.332625 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:04.332603 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d_odh-ai-gateway-infra(8bfe2e39-f860-49e7-9fab-2694c2e85334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" Nov 03 20:15:04.340587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.340567 5815 scope.go:122] "RemoveContainer" containerID="ffae04c02238e1bab3f01af8fc3eaaa2cc4abb07afe9d82494d00573ef69e6a1" Nov 03 20:15:04.632881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:04.632803 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:15:05.347049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:05.347023 5815 scope.go:122] "RemoveContainer" containerID="4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681" Nov 03 20:15:05.347467 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:05.347257 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-x8dnj_odh-ai-gateway-infra(99c5cf8e-2afb-4ed6-9944-978c3cd59d6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" Nov 03 20:15:05.349861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:05.349838 5815 scope.go:122] "RemoveContainer" containerID="e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c" Nov 03 20:15:05.350145 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:05.350121 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d_odh-ai-gateway-infra(8bfe2e39-f860-49e7-9fab-2694c2e85334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" Nov 03 20:15:08.950716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:08.950618 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:08.951395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:08.951349 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:09.424948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.424917 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:09.478649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.478612 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:09.478937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.478909 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:09.495669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.495639 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:09.495913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.495892 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:09.734904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.734877 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:09.917531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:09.914754 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:13.578081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:13.578049 5815 scope.go:122] "RemoveContainer" containerID="4a9872f67d204509acb3a7642ec5be6f4202bfa5ef2586ebc9dfe1daff6fa4a4" Nov 03 20:15:13.854577 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:13.854458 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:15:14.501420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:14.501383 5815 generic.go:356] "Generic (PLEG): container finished" podID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerID="7f367faa918b766dc83337d5a4bb500dd86f4479805d43308a008e0e8fbe6d79" exitCode=60 Nov 03 20:15:14.501420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:14.501412 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" event={"ID":"4e2d3584-09d0-4549-9735-dcafcf867c0e","Type":"ContainerDied","Data":"7f367faa918b766dc83337d5a4bb500dd86f4479805d43308a008e0e8fbe6d79"} Nov 03 20:15:14.501681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:14.501490 5815 scope.go:122] "RemoveContainer" containerID="4a9872f67d204509acb3a7642ec5be6f4202bfa5ef2586ebc9dfe1daff6fa4a4" Nov 03 20:15:14.501930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:14.501895 5815 scope.go:122] "RemoveContainer" containerID="7f367faa918b766dc83337d5a4bb500dd86f4479805d43308a008e0e8fbe6d79" Nov 03 20:15:14.502234 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:14.502209 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl_odh-ai-gateway-infra(4e2d3584-09d0-4549-9735-dcafcf867c0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" Nov 03 20:15:14.578707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:14.578684 5815 scope.go:122] "RemoveContainer" containerID="e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c" Nov 03 20:15:14.579144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:14.578935 5815 scope.go:122] "RemoveContainer" containerID="4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681" Nov 03 20:15:15.524569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.522856 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerStarted","Data":"f303408b2375b14c638255eb4b7683c7a4da90ab0e9c59cd2a00b8fcee9985bc"} Nov 03 20:15:15.529316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.529288 5815 generic.go:356] "Generic (PLEG): container finished" podID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerID="9150913190b1d710fe7ff47000284409205ccb8bc4c5bffe29ef23e5e2950f3a" exitCode=60 Nov 03 20:15:15.529488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.529335 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerDied","Data":"9150913190b1d710fe7ff47000284409205ccb8bc4c5bffe29ef23e5e2950f3a"} Nov 03 20:15:15.529488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.529404 5815 scope.go:122] "RemoveContainer" containerID="4c3bacc92d4622c2d56f75358bf13c3ccfc2914b60e661f7f2f6837f25f01681" Nov 03 20:15:15.529865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.529838 5815 scope.go:122] "RemoveContainer" containerID="9150913190b1d710fe7ff47000284409205ccb8bc4c5bffe29ef23e5e2950f3a" Nov 03 20:15:15.530289 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:15.530204 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895615-x8dnj_odh-ai-gateway-infra(99c5cf8e-2afb-4ed6-9944-978c3cd59d6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" Nov 03 20:15:15.562013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.561989 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl"] Nov 03 20:15:15.577878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.577858 5815 scope.go:122] "RemoveContainer" containerID="4c41a40810fb571c03e92a522281140984b6349a5d610454efac9cce947ce173" Nov 03 20:15:15.738033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.737974 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:15.834831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.834802 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l\" (UniqueName: \"kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l\") pod \"4e2d3584-09d0-4549-9735-dcafcf867c0e\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " Nov 03 20:15:15.834998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.834984 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca\") pod \"4e2d3584-09d0-4549-9735-dcafcf867c0e\" (UID: \"4e2d3584-09d0-4549-9735-dcafcf867c0e\") " Nov 03 20:15:15.835313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.835285 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca" pod "4e2d3584-09d0-4549-9735-dcafcf867c0e" (UID: "4e2d3584-09d0-4549-9735-dcafcf867c0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:15.837186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.837161 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l" pod "4e2d3584-09d0-4549-9735-dcafcf867c0e" (UID: "4e2d3584-09d0-4549-9735-dcafcf867c0e"). InnerVolumeSpecName "kube-api-access-67f2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:15.935946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.935922 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67f2l\" (UniqueName: \"kubernetes.io/projected/4e2d3584-09d0-4549-9735-dcafcf867c0e-kube-api-access-67f2l\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:15.935946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:15.935947 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2d3584-09d0-4549-9735-dcafcf867c0e-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:16.538934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.538905 5815 generic.go:356] "Generic (PLEG): container finished" podID="5079d318-0304-43b6-bf50-d517c1b9b217" containerID="6e8c4bf117c70df1e7b553690c9b76e816169f17c30817d330a04494417a9f07" exitCode=60 Nov 03 20:15:16.539101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.538978 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" event={"ID":"5079d318-0304-43b6-bf50-d517c1b9b217","Type":"ContainerDied","Data":"6e8c4bf117c70df1e7b553690c9b76e816169f17c30817d330a04494417a9f07"} Nov 03 20:15:16.539101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.539089 5815 scope.go:122] "RemoveContainer" containerID="4c41a40810fb571c03e92a522281140984b6349a5d610454efac9cce947ce173" Nov 03 20:15:16.539560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.539532 5815 scope.go:122] "RemoveContainer" containerID="6e8c4bf117c70df1e7b553690c9b76e816169f17c30817d330a04494417a9f07" Nov 03 20:15:16.539909 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:16.539879 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m_odh-ai-gateway-infra(5079d318-0304-43b6-bf50-d517c1b9b217)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" Nov 03 20:15:16.553114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.541234 5815 generic.go:356] "Generic (PLEG): container finished" podID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerID="f303408b2375b14c638255eb4b7683c7a4da90ab0e9c59cd2a00b8fcee9985bc" exitCode=60 Nov 03 20:15:16.553114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.541301 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerDied","Data":"f303408b2375b14c638255eb4b7683c7a4da90ab0e9c59cd2a00b8fcee9985bc"} Nov 03 20:15:16.553114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.541777 5815 scope.go:122] "RemoveContainer" containerID="f303408b2375b14c638255eb4b7683c7a4da90ab0e9c59cd2a00b8fcee9985bc" Nov 03 20:15:16.553114 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:16.542025 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d_odh-ai-gateway-infra(8bfe2e39-f860-49e7-9fab-2694c2e85334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" Nov 03 20:15:16.553114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.542987 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" event={"ID":"4e2d3584-09d0-4549-9735-dcafcf867c0e","Type":"ContainerDied","Data":"13530643366d94a463c00b01c2cf576c37c9c9756aff1d4058c9cfcda0dbf048"} Nov 03 20:15:16.553114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.543063 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl" Nov 03 20:15:16.553771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.553750 5815 scope.go:122] "RemoveContainer" containerID="e4f3bdbbf1c3f95ae774cb3f21c69a9480027d24238f456a26a28e44a2b5f50c" Nov 03 20:15:16.568211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.568191 5815 scope.go:122] "RemoveContainer" containerID="7f367faa918b766dc83337d5a4bb500dd86f4479805d43308a008e0e8fbe6d79" Nov 03 20:15:16.571260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.571236 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl"] Nov 03 20:15:16.587302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.587279 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895615-9wfnl"] Nov 03 20:15:16.587402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.587343 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d"] Nov 03 20:15:16.651776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.651741 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj"] Nov 03 20:15:16.800931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.800906 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:16.844482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.844454 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca\") pod \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " Nov 03 20:15:16.844698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.844541 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c\" (UniqueName: \"kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c\") pod \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\" (UID: \"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a\") " Nov 03 20:15:16.845144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.845107 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca" pod "99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" (UID: "99c5cf8e-2afb-4ed6-9944-978c3cd59d6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:16.847590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.847562 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c" pod "99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" (UID: "99c5cf8e-2afb-4ed6-9944-978c3cd59d6a"). InnerVolumeSpecName "kube-api-access-vbt7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:16.945628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.945594 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:16.945628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:16.945622 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbt7c\" (UniqueName: \"kubernetes.io/projected/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a-kube-api-access-vbt7c\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.110084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.109991 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x"] Nov 03 20:15:17.558384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.558352 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" event={"ID":"99c5cf8e-2afb-4ed6-9944-978c3cd59d6a","Type":"ContainerDied","Data":"6dfc9711fc4585e8045571c29e39239b934b6145923d2e394d79df0f1604ca60"} Nov 03 20:15:17.558520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.558413 5815 scope.go:122] "RemoveContainer" containerID="9150913190b1d710fe7ff47000284409205ccb8bc4c5bffe29ef23e5e2950f3a" Nov 03 20:15:17.558520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.558365 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj" Nov 03 20:15:17.570446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.570382 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m"] Nov 03 20:15:17.598052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.598010 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj"] Nov 03 20:15:17.602916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.602865 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895615-x8dnj"] Nov 03 20:15:17.828556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.828532 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:17.830929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.830913 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:17.861127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861098 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca\") pod \"8bfe2e39-f860-49e7-9fab-2694c2e85334\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " Nov 03 20:15:17.861281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861149 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8\" (UniqueName: \"kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8\") pod \"5079d318-0304-43b6-bf50-d517c1b9b217\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " Nov 03 20:15:17.861281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861247 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg\" (UniqueName: \"kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg\") pod \"8bfe2e39-f860-49e7-9fab-2694c2e85334\" (UID: \"8bfe2e39-f860-49e7-9fab-2694c2e85334\") " Nov 03 20:15:17.861405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861322 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca\") pod \"5079d318-0304-43b6-bf50-d517c1b9b217\" (UID: \"5079d318-0304-43b6-bf50-d517c1b9b217\") " Nov 03 20:15:17.861547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861519 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca" pod "8bfe2e39-f860-49e7-9fab-2694c2e85334" (UID: "8bfe2e39-f860-49e7-9fab-2694c2e85334"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:17.861893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861779 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bfe2e39-f860-49e7-9fab-2694c2e85334-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.861991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.861966 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca" pod "5079d318-0304-43b6-bf50-d517c1b9b217" (UID: "5079d318-0304-43b6-bf50-d517c1b9b217"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:17.864362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.864340 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8" pod "5079d318-0304-43b6-bf50-d517c1b9b217" (UID: "5079d318-0304-43b6-bf50-d517c1b9b217"). InnerVolumeSpecName "kube-api-access-h7qv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:17.864698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.864677 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg" pod "8bfe2e39-f860-49e7-9fab-2694c2e85334" (UID: "8bfe2e39-f860-49e7-9fab-2694c2e85334"). InnerVolumeSpecName "kube-api-access-bnkmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:17.963013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.962983 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5079d318-0304-43b6-bf50-d517c1b9b217-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.963013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.963007 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7qv8\" (UniqueName: \"kubernetes.io/projected/5079d318-0304-43b6-bf50-d517c1b9b217-kube-api-access-h7qv8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.963013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:17.963017 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkmg\" (UniqueName: \"kubernetes.io/projected/8bfe2e39-f860-49e7-9fab-2694c2e85334-kube-api-access-bnkmg\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.569051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.569017 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" Nov 03 20:15:18.569256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.569207 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m" event={"ID":"5079d318-0304-43b6-bf50-d517c1b9b217","Type":"ContainerDied","Data":"82a584ecbc2ec9f12b654a060a678b49a499a14d944772d35a4539cfcaf3b2b7"} Nov 03 20:15:18.569320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.569273 5815 scope.go:122] "RemoveContainer" containerID="6e8c4bf117c70df1e7b553690c9b76e816169f17c30817d330a04494417a9f07" Nov 03 20:15:18.571395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.571365 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" event={"ID":"8bfe2e39-f860-49e7-9fab-2694c2e85334","Type":"ContainerDied","Data":"080505df66089462f257ef5758c4983b21ee14da33745c321ff04eceeeea48ae"} Nov 03 20:15:18.571572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.571466 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d" Nov 03 20:15:18.585198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.585169 5815 scope.go:122] "RemoveContainer" containerID="f303408b2375b14c638255eb4b7683c7a4da90ab0e9c59cd2a00b8fcee9985bc" Nov 03 20:15:18.602898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.602868 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" path="/var/lib/kubelet/pods/4e2d3584-09d0-4549-9735-dcafcf867c0e/volumes" Nov 03 20:15:18.603880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.603556 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" path="/var/lib/kubelet/pods/99c5cf8e-2afb-4ed6-9944-978c3cd59d6a/volumes" Nov 03 20:15:18.607103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.607083 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m"] Nov 03 20:15:18.609825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.609804 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895615-htc6m"] Nov 03 20:15:18.618067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.618046 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d"] Nov 03 20:15:18.623192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:18.623174 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895615-ndc6d"] Nov 03 20:15:20.588804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:20.588768 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" path="/var/lib/kubelet/pods/5079d318-0304-43b6-bf50-d517c1b9b217/volumes" Nov 03 20:15:20.589339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:20.589314 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" path="/var/lib/kubelet/pods/8bfe2e39-f860-49e7-9fab-2694c2e85334/volumes" Nov 03 20:15:23.797946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.797902 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2"] Nov 03 20:15:23.798602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798585 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.798658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798606 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.798658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798621 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.798658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798629 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.798658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798643 5815 cpu_manager.go:414] "containerMap: removing container" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:23.798658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798650 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798668 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798675 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798692 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798699 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798722 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798729 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798742 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798749 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798760 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798766 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798779 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798786 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798796 5815 cpu_manager.go:414] "containerMap: removing container" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:23.798825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798802 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798930 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798941 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798948 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798956 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798964 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798972 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798980 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798990 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.798997 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2d3584-09d0-4549-9735-dcafcf867c0e" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.799008 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bfe2e39-f860-49e7-9fab-2694c2e85334" containerName="cleanup" Nov 03 20:15:23.799251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.799015 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:23.805798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.805149 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:23.809723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.809672 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"distinct-w3-4f93d1-kserve-self-signed-certs\"" Nov 03 20:15:23.812024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.811983 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2"] Nov 03 20:15:23.930683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.930198 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x662h\" (UniqueName: \"kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:23.930683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.930263 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:23.930683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.930313 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:23.930683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.930334 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:23.930683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.930400 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:23.930683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:23.930439 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.031941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.031900 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.031941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.031940 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.031986 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.032013 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.032127 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x662h\" (UniqueName: \"kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.032179 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.032323 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.032333 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.032794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.032750 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.034464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.034447 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.034839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.034820 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.044384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.044362 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x662h\" (UniqueName: \"kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h\") pod \"distinct-w3-4f93d1-kserve-945b8987-7qhs2\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.122599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.122483 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:24.409068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.409044 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2"] Nov 03 20:15:24.410932 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:24.410905 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e15d366_3ba2_487e_8c5e_0d0cd4b65040.slice/crio-50c36e3b16efc6e11004a32c241039722f9842eb31e6e2986031fb629d94cefe WatchSource:0}: Error finding container 50c36e3b16efc6e11004a32c241039722f9842eb31e6e2986031fb629d94cefe: Status 404 returned error can't find the container with id 50c36e3b16efc6e11004a32c241039722f9842eb31e6e2986031fb629d94cefe Nov 03 20:15:24.574974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.574941 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h"] Nov 03 20:15:24.575400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.575386 5815 cpu_manager.go:414] "containerMap: removing container" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:24.575400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.575400 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:24.575569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.575436 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:24.575569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.575441 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5079d318-0304-43b6-bf50-d517c1b9b217" containerName="cleanup" Nov 03 20:15:24.575569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.575537 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c5cf8e-2afb-4ed6-9944-978c3cd59d6a" containerName="cleanup" Nov 03 20:15:24.579015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.578989 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.594623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.594600 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4f93d1-openshift-default-dockercfg-kqdcn\"" Nov 03 20:15:24.598631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.598607 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h"] Nov 03 20:15:24.629706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.629676 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" event={"ID":"6e15d366-3ba2-487e-8c5e-0d0cd4b65040","Type":"ContainerStarted","Data":"8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17"} Nov 03 20:15:24.629706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.629712 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" event={"ID":"6e15d366-3ba2-487e-8c5e-0d0cd4b65040","Type":"ContainerStarted","Data":"50c36e3b16efc6e11004a32c241039722f9842eb31e6e2986031fb629d94cefe"} Nov 03 20:15:24.640837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.640808 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.640956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.640940 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.640972 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.640997 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.641089 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.641116 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.641145 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.641251 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.641321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.641274 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzppg\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.656913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.656875 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podStartSLOduration=1.656863859 podStartE2EDuration="1.656863859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:24.655392912 +0000 UTC m=+834.185738584" watchObservedRunningTime="2026-11-03 20:15:24.656863859 +0000 UTC m=+834.187209531" Nov 03 20:15:24.742727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742700 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.742883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742743 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.742883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742779 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.742883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742814 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.742883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742843 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742888 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742940 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.742969 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzppg\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.743069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.743235 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.743712 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.743902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.743881 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.744039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.744019 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.744097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.744076 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.745154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.745134 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.745432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.745415 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.755768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.755742 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.755956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.755938 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzppg\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:24.896551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:24.896520 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:25.123241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.123167 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:25.186630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.186599 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h"] Nov 03 20:15:25.188185 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:25.188156 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35db10e6_9717_4145_be19_db5022ebfe24.slice/crio-0d3ba13ab745fa15f8b026805c7530874a0af022e1582290aa8f509f60914ae4 WatchSource:0}: Error finding container 0d3ba13ab745fa15f8b026805c7530874a0af022e1582290aa8f509f60914ae4: Status 404 returned error can't find the container with id 0d3ba13ab745fa15f8b026805c7530874a0af022e1582290aa8f509f60914ae4 Nov 03 20:15:25.192079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.192040 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:15:25.192197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.192109 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:15:25.192197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.192158 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:15:25.640161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.640122 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" event={"ID":"35db10e6-9717-4145-be19-db5022ebfe24","Type":"ContainerStarted","Data":"0e33d1677f3e18f1724062f459bdc18edcc9d6fd5a13b7f3cfd7bc3615b4be57"} Nov 03 20:15:25.640346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.640183 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" event={"ID":"35db10e6-9717-4145-be19-db5022ebfe24","Type":"ContainerStarted","Data":"0d3ba13ab745fa15f8b026805c7530874a0af022e1582290aa8f509f60914ae4"} Nov 03 20:15:25.668005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.667934 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podStartSLOduration=1.6679120649999999 podStartE2EDuration="1.667912065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:25.663367691 +0000 UTC m=+835.193713363" watchObservedRunningTime="2026-11-03 20:15:25.667912065 +0000 UTC m=+835.198257738" Nov 03 20:15:25.896764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:25.896685 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:26.899379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:26.899338 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:15:26.899823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:26.899402 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:15:27.897821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:27.897779 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:15:27.898008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:27.897848 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:15:28.232094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.232064 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2"] Nov 03 20:15:28.236240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.236215 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.242830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.242810 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:28.243179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.243161 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:15:28.245802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.245649 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2"] Nov 03 20:15:28.307531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.307477 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.307884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.307848 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.308014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.307894 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnt2b\" (UniqueName: \"kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.411955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.411913 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.412121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.412016 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.412121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.412048 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnt2b\" (UniqueName: \"kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.413585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.413555 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.414922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.414892 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7"] Nov 03 20:15:28.416986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.416638 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.420023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.419879 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.446577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.446489 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7"] Nov 03 20:15:28.455090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.455071 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnt2b\" (UniqueName: \"kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b\") pod \"maas-api-revoke-sub-8a50d7ad-z9zc2\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.513737 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.513864 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.513893 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.514072 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.514146 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.514174 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.514243 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.514444 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.514580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.514563 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm8vc\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.557117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.557085 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:28.619203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619152 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619215 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619250 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619292 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619397 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619444 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8vc\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619572 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619600 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.619753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.619624 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.620356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.620057 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.651007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.621412 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.651007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.622058 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.651007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.632864 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.651007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.634099 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.651007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.645186 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.656573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.652441 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8vc\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.663212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.663180 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.676537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.675637 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.837640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.837291 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:28.917241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.907001 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:15:28.917241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:28.907057 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:15:29.161449 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:29.161416 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1741ef8_0dd2_48a2_b41b_e2576488eeb3.slice/crio-b0d421c0707b8e676f70aa7f93125dffa58adf099109bd8928f4953bd0ffcaa5 WatchSource:0}: Error finding container b0d421c0707b8e676f70aa7f93125dffa58adf099109bd8928f4953bd0ffcaa5: Status 404 returned error can't find the container with id b0d421c0707b8e676f70aa7f93125dffa58adf099109bd8928f4953bd0ffcaa5 Nov 03 20:15:29.162202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.162176 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2"] Nov 03 20:15:29.257365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.257343 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7"] Nov 03 20:15:29.258268 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:29.258245 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c35e48_442e_4a6b_8796_3d086c92b955.slice/crio-62a5538a62168459eebe340b25e59cee5ce147d8ba32f25ca2b0de9456b96fae WatchSource:0}: Error finding container 62a5538a62168459eebe340b25e59cee5ce147d8ba32f25ca2b0de9456b96fae: Status 404 returned error can't find the container with id 62a5538a62168459eebe340b25e59cee5ce147d8ba32f25ca2b0de9456b96fae Nov 03 20:15:29.260566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.260485 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:15:29.260642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.260611 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:15:29.260701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.260646 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:15:29.717920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.717873 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" event={"ID":"e1741ef8-0dd2-48a2-b41b-e2576488eeb3","Type":"ContainerStarted","Data":"cd7627fc80254aac1ab64f073d7a88b4f2050cd7f861fd839d082d5356181834"} Nov 03 20:15:29.718093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.717985 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" event={"ID":"e1741ef8-0dd2-48a2-b41b-e2576488eeb3","Type":"ContainerStarted","Data":"b0d421c0707b8e676f70aa7f93125dffa58adf099109bd8928f4953bd0ffcaa5"} Nov 03 20:15:29.720617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.720589 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" event={"ID":"e6c35e48-442e-4a6b-8796-3d086c92b955","Type":"ContainerStarted","Data":"e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594"} Nov 03 20:15:29.720720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.720642 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" event={"ID":"e6c35e48-442e-4a6b-8796-3d086c92b955","Type":"ContainerStarted","Data":"62a5538a62168459eebe340b25e59cee5ce147d8ba32f25ca2b0de9456b96fae"} Nov 03 20:15:29.737432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.737377 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" podStartSLOduration=1.737361551 podStartE2EDuration="1.737361551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:29.736049919 +0000 UTC m=+839.266395586" watchObservedRunningTime="2026-11-03 20:15:29.737361551 +0000 UTC m=+839.267707224" Nov 03 20:15:29.763225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.763176 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podStartSLOduration=1.7631599260000002 podStartE2EDuration="1.763159926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:29.761228016 +0000 UTC m=+839.291573689" watchObservedRunningTime="2026-11-03 20:15:29.763159926 +0000 UTC m=+839.293505599" Nov 03 20:15:29.837951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.837921 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:29.900308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.899831 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:15:29.900308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:29.899888 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:15:30.746374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:30.745222 5815 generic.go:356] "Generic (PLEG): container finished" podID="e1741ef8-0dd2-48a2-b41b-e2576488eeb3" containerID="cd7627fc80254aac1ab64f073d7a88b4f2050cd7f861fd839d082d5356181834" exitCode=0 Nov 03 20:15:30.746374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:30.746122 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" event={"ID":"e1741ef8-0dd2-48a2-b41b-e2576488eeb3","Type":"ContainerDied","Data":"cd7627fc80254aac1ab64f073d7a88b4f2050cd7f861fd839d082d5356181834"} Nov 03 20:15:30.841644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:30.841623 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:15:30.841811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:30.841790 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:15:30.902585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:30.900025 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:15:30.902585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:30.900080 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:15:31.843141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:31.840558 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:15:31.843141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:31.840625 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:15:32.111533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.100766 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:32.137420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.134653 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" podUID="a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" containerName="main" containerID="cri-o://b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5" gracePeriod=45 Nov 03 20:15:32.147664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.147633 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b\" (UniqueName: \"kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b\") pod \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " Nov 03 20:15:32.147784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.147763 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca\") pod \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " Nov 03 20:15:32.147868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.147823 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca\") pod \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\" (UID: \"e1741ef8-0dd2-48a2-b41b-e2576488eeb3\") " Nov 03 20:15:32.149147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.149110 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca" pod "e1741ef8-0dd2-48a2-b41b-e2576488eeb3" (UID: "e1741ef8-0dd2-48a2-b41b-e2576488eeb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:32.150896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.150873 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca" pod "e1741ef8-0dd2-48a2-b41b-e2576488eeb3" (UID: "e1741ef8-0dd2-48a2-b41b-e2576488eeb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:32.152543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.152376 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b" pod "e1741ef8-0dd2-48a2-b41b-e2576488eeb3" (UID: "e1741ef8-0dd2-48a2-b41b-e2576488eeb3"). InnerVolumeSpecName "kube-api-access-lnt2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:32.249994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.249956 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnt2b\" (UniqueName: \"kubernetes.io/projected/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-kube-api-access-lnt2b\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.249994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.249997 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.250247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.250013 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1741ef8-0dd2-48a2-b41b-e2576488eeb3-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.251987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.251947 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:32.252755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.252729 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:32.254180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.254157 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:15:32.511435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.511403 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:15:32.564423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.564390 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home\") pod \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " Nov 03 20:15:32.564636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.564434 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm\") pod \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " Nov 03 20:15:32.564636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.564487 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9\" (UniqueName: \"kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9\") pod \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " Nov 03 20:15:32.581531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.568860 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm" pod "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" (UID: "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:32.581531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.572578 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs\") pod \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " Nov 03 20:15:32.581531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.572677 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir\") pod \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " Nov 03 20:15:32.581531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.572751 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache\") pod \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\" (UID: \"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd\") " Nov 03 20:15:32.581531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.573163 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.581531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.576031 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir" pod "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" (UID: "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:32.582601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.582572 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache" pod "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" (UID: "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:32.589526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.582988 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9" pod "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" (UID: "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd"). InnerVolumeSpecName "kube-api-access-vnbk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:32.589644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.589573 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home" pod "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" (UID: "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:32.603770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.603693 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs" pod "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" (UID: "a0ca2df6-5587-4d4e-b59e-48b9f7d326dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:32.686531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.674233 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.686531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.674269 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnbk9\" (UniqueName: \"kubernetes.io/projected/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-kube-api-access-vnbk9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.686531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.674287 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.686531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.674300 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.686531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.674313 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:32.760690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.760656 5815 generic.go:356] "Generic (PLEG): container finished" podID="a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" containerID="b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5" exitCode=0 Nov 03 20:15:32.760874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.760707 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" Nov 03 20:15:32.760874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.760739 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" event={"ID":"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd","Type":"ContainerDied","Data":"b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5"} Nov 03 20:15:32.760874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.760798 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x" event={"ID":"a0ca2df6-5587-4d4e-b59e-48b9f7d326dd","Type":"ContainerDied","Data":"cffdc54a41cc67301d6a57e3ad2d7f9106e96a5cd75b8e748f00bb56468b8ece"} Nov 03 20:15:32.760874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.760827 5815 scope.go:122] "RemoveContainer" containerID="b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5" Nov 03 20:15:32.762985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.762962 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" Nov 03 20:15:32.763099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.763007 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2" event={"ID":"e1741ef8-0dd2-48a2-b41b-e2576488eeb3","Type":"ContainerDied","Data":"b0d421c0707b8e676f70aa7f93125dffa58adf099109bd8928f4953bd0ffcaa5"} Nov 03 20:15:32.763099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.763050 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0d421c0707b8e676f70aa7f93125dffa58adf099109bd8928f4953bd0ffcaa5" Nov 03 20:15:32.812606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.812585 5815 scope.go:122] "RemoveContainer" containerID="b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5" Nov 03 20:15:32.812982 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:32.812958 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5\": container with ID starting with b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5 not found: ID does not exist" containerID="b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5" Nov 03 20:15:32.813111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.812993 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5"} err="failed to get container status \"b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5\": rpc error: code = NotFound desc = could not find container \"b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5\": container with ID starting with b408fee442f51857a200c3aa868a7f27e248aa5d16f7f7a43d575476d63216b5 not found: ID does not exist" Nov 03 20:15:32.815605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.815578 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x"] Nov 03 20:15:32.819925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.819904 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-a-256c5a80-kserve-586c67d8ff-9rs5x"] Nov 03 20:15:32.838279 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.838258 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:15:32.838364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:32.838299 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:15:33.838604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:33.838569 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:15:33.839005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:33.838621 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:15:34.123028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:34.122937 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:34.135762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:34.135733 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:34.600338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:34.600295 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" path="/var/lib/kubelet/pods/a0ca2df6-5587-4d4e-b59e-48b9f7d326dd/volumes" Nov 03 20:15:34.826897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:34.826870 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:34.838358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:34.838329 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:15:34.838497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:34.838367 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:15:35.364528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:35.364473 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:15:35.838712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:35.838682 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:15:35.839035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:35.838735 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Nov 03 20:15:37.423484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:37.423449 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:37.425204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:37.425184 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:37.446531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:37.443805 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 20:15:37.827672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:37.827630 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:15:38.059533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:38.058614 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:38.361691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:38.361639 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:38.361999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:38.361949 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.061531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:39.060590 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:15:39.061531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:39.061212 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:15:39.777541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:39.777364 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.850533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:39.825605 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.852183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:39.851223 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.064735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.064622 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.174665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.174605 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:15:40.557543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.532670 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.804645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.794696 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.829271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.828614 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:15:40.853715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.853644 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.988520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.988469 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:40.997531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.993631 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:15:40.997831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:40.997813 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:41.086587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:41.071870 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:15:41.175586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:41.175538 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:41.177912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:41.177888 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:15:41.469603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:41.469333 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:15:41.954326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:41.954232 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:15:42.003556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.003358 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:15:42.005098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.005062 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:42.055939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.055908 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg"] Nov 03 20:15:42.082327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.082296 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e1741ef8-0dd2-48a2-b41b-e2576488eeb3" containerName="revoke-keys" Nov 03 20:15:42.082543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.082526 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1741ef8-0dd2-48a2-b41b-e2576488eeb3" containerName="revoke-keys" Nov 03 20:15:42.082714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.082702 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" containerName="main" Nov 03 20:15:42.082809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.082797 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" containerName="main" Nov 03 20:15:42.083044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.083028 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ca2df6-5587-4d4e-b59e-48b9f7d326dd" containerName="main" Nov 03 20:15:42.083166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.083154 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1741ef8-0dd2-48a2-b41b-e2576488eeb3" containerName="revoke-keys" Nov 03 20:15:42.168540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.144395 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:42.276218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.276188 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg"] Nov 03 20:15:42.312184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.298642 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.312670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.312647 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:42.335580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.313074 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:15:42.335749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.331786 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:15:42.571574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.569417 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tjk\" (UniqueName: \"kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.571574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.569688 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.571574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.569746 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.636532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.635564 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:15:42.651388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.651364 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:42.689180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.689141 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.689938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.689912 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.691653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.690760 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.691851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.691826 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tjk\" (UniqueName: \"kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.714085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.714022 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.763123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.762944 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tjk\" (UniqueName: \"kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk\") pod \"maas-api-revoke-sub-d33a5ba4-dxdgg\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:42.785140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:42.784885 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:43.120168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.120137 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg"] Nov 03 20:15:43.121720 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:43.121698 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997f91a2_55e4_4c99_b40a_fabd64ee5d44.slice/crio-42569a9f23a61a6838238f7076af6be715ea24d32552f51825e3caeea35a9f20 WatchSource:0}: Error finding container 42569a9f23a61a6838238f7076af6be715ea24d32552f51825e3caeea35a9f20: Status 404 returned error can't find the container with id 42569a9f23a61a6838238f7076af6be715ea24d32552f51825e3caeea35a9f20 Nov 03 20:15:43.535202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.535166 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk"] Nov 03 20:15:43.538938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.538920 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.548633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.548606 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk"] Nov 03 20:15:43.621917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.621885 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7kv\" (UniqueName: \"kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.622100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.621961 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.622100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.622064 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.723058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.723022 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7kv\" (UniqueName: \"kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.723233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.723098 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.723233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.723134 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.724016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.723986 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.725779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.725752 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.735803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.735779 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7kv\" (UniqueName: \"kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv\") pod \"maas-api-revoke-sub-b8e6cc3e-pfsbk\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.870556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:43.869840 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:43.870556 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:15:43.870085 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:15:44.015213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:44.015032 5815 generic.go:356] "Generic (PLEG): container finished" podID="997f91a2-55e4-4c99-b40a-fabd64ee5d44" containerID="1cdc570b7f905d3f42bc4e63833cf6662e1aaab12fed0791f35239a9ebbe12e2" exitCode=0 Nov 03 20:15:44.015213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:44.015090 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" event={"ID":"997f91a2-55e4-4c99-b40a-fabd64ee5d44","Type":"ContainerDied","Data":"1cdc570b7f905d3f42bc4e63833cf6662e1aaab12fed0791f35239a9ebbe12e2"} Nov 03 20:15:44.015213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:44.015173 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" event={"ID":"997f91a2-55e4-4c99-b40a-fabd64ee5d44","Type":"ContainerStarted","Data":"42569a9f23a61a6838238f7076af6be715ea24d32552f51825e3caeea35a9f20"} Nov 03 20:15:44.195243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:44.195218 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk"] Nov 03 20:15:44.196238 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:44.196201 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc17f05_6452_45c0_a405_19f71acfc3d6.slice/crio-eb1da812487b9eeee0aed570d6e71f38799cdfe2ca692f130084f009af671110 WatchSource:0}: Error finding container eb1da812487b9eeee0aed570d6e71f38799cdfe2ca692f130084f009af671110: Status 404 returned error can't find the container with id eb1da812487b9eeee0aed570d6e71f38799cdfe2ca692f130084f009af671110 Nov 03 20:15:45.023744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.023715 5815 generic.go:356] "Generic (PLEG): container finished" podID="9fc17f05-6452-45c0-a405-19f71acfc3d6" containerID="92faaef97786a5cf1cdf055b71a1a944184ea7043533d569039cf922034f0294" exitCode=0 Nov 03 20:15:45.024140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.023800 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" event={"ID":"9fc17f05-6452-45c0-a405-19f71acfc3d6","Type":"ContainerDied","Data":"92faaef97786a5cf1cdf055b71a1a944184ea7043533d569039cf922034f0294"} Nov 03 20:15:45.024140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.023851 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" event={"ID":"9fc17f05-6452-45c0-a405-19f71acfc3d6","Type":"ContainerStarted","Data":"eb1da812487b9eeee0aed570d6e71f38799cdfe2ca692f130084f009af671110"} Nov 03 20:15:45.193197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.193171 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:45.250990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.249649 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk\" (UniqueName: \"kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk\") pod \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " Nov 03 20:15:45.250990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.249711 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca\") pod \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " Nov 03 20:15:45.250990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.249784 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca\") pod \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\" (UID: \"997f91a2-55e4-4c99-b40a-fabd64ee5d44\") " Nov 03 20:15:45.250990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.250135 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca" pod "997f91a2-55e4-4c99-b40a-fabd64ee5d44" (UID: "997f91a2-55e4-4c99-b40a-fabd64ee5d44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:45.252618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.252555 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk" pod "997f91a2-55e4-4c99-b40a-fabd64ee5d44" (UID: "997f91a2-55e4-4c99-b40a-fabd64ee5d44"). InnerVolumeSpecName "kube-api-access-n5tjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:45.256482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.256454 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca" pod "997f91a2-55e4-4c99-b40a-fabd64ee5d44" (UID: "997f91a2-55e4-4c99-b40a-fabd64ee5d44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:45.351236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.351205 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5tjk\" (UniqueName: \"kubernetes.io/projected/997f91a2-55e4-4c99-b40a-fabd64ee5d44-kube-api-access-n5tjk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:45.351236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.351233 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:45.351460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:45.351243 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997f91a2-55e4-4c99-b40a-fabd64ee5d44-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:46.031997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.031956 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" event={"ID":"997f91a2-55e4-4c99-b40a-fabd64ee5d44","Type":"ContainerDied","Data":"42569a9f23a61a6838238f7076af6be715ea24d32552f51825e3caeea35a9f20"} Nov 03 20:15:46.031997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.031994 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg" Nov 03 20:15:46.032527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.032011 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42569a9f23a61a6838238f7076af6be715ea24d32552f51825e3caeea35a9f20" Nov 03 20:15:46.188040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.188016 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:46.261029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.261003 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca\") pod \"9fc17f05-6452-45c0-a405-19f71acfc3d6\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " Nov 03 20:15:46.261201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.261035 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca\") pod \"9fc17f05-6452-45c0-a405-19f71acfc3d6\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " Nov 03 20:15:46.261201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.261069 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv\" (UniqueName: \"kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv\") pod \"9fc17f05-6452-45c0-a405-19f71acfc3d6\" (UID: \"9fc17f05-6452-45c0-a405-19f71acfc3d6\") " Nov 03 20:15:46.261434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.261410 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca" pod "9fc17f05-6452-45c0-a405-19f71acfc3d6" (UID: "9fc17f05-6452-45c0-a405-19f71acfc3d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:46.263458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.263428 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv" pod "9fc17f05-6452-45c0-a405-19f71acfc3d6" (UID: "9fc17f05-6452-45c0-a405-19f71acfc3d6"). InnerVolumeSpecName "kube-api-access-cx7kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:46.263458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.263449 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca" pod "9fc17f05-6452-45c0-a405-19f71acfc3d6" (UID: "9fc17f05-6452-45c0-a405-19f71acfc3d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:46.362549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.362465 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:46.362549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.362491 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc17f05-6452-45c0-a405-19f71acfc3d6-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:46.362549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:46.362500 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7kv\" (UniqueName: \"kubernetes.io/projected/9fc17f05-6452-45c0-a405-19f71acfc3d6-kube-api-access-cx7kv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:47.041649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:47.041592 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" event={"ID":"9fc17f05-6452-45c0-a405-19f71acfc3d6","Type":"ContainerDied","Data":"eb1da812487b9eeee0aed570d6e71f38799cdfe2ca692f130084f009af671110"} Nov 03 20:15:47.041649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:47.041623 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk" Nov 03 20:15:47.041649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:47.041639 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb1da812487b9eeee0aed570d6e71f38799cdfe2ca692f130084f009af671110" Nov 03 20:15:52.205584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.205547 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm"] Nov 03 20:15:52.206253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.206232 5815 cpu_manager.go:414] "containerMap: removing container" podUID="997f91a2-55e4-4c99-b40a-fabd64ee5d44" containerName="revoke-keys" Nov 03 20:15:52.206366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.206254 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997f91a2-55e4-4c99-b40a-fabd64ee5d44" containerName="revoke-keys" Nov 03 20:15:52.206366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.206272 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9fc17f05-6452-45c0-a405-19f71acfc3d6" containerName="revoke-keys" Nov 03 20:15:52.206366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.206279 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc17f05-6452-45c0-a405-19f71acfc3d6" containerName="revoke-keys" Nov 03 20:15:52.206533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.206473 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="997f91a2-55e4-4c99-b40a-fabd64ee5d44" containerName="revoke-keys" Nov 03 20:15:52.206533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.206488 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc17f05-6452-45c0-a405-19f71acfc3d6" containerName="revoke-keys" Nov 03 20:15:52.210058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.210029 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.215769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.215742 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:52.215931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.215908 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:15:52.220741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.220718 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm"] Nov 03 20:15:52.327576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.327543 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tr7\" (UniqueName: \"kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.327739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.327606 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.327739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.327691 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.428922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.428893 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.429111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.428943 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.429111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.429051 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tr7\" (UniqueName: \"kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.429628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.429608 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.431369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.431348 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.441427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.441405 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tr7\" (UniqueName: \"kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7\") pod \"maas-api-revoke-sub-e2b5f8cc-dxjsm\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.527434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.527406 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:52.841110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:52.840924 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm"] Nov 03 20:15:52.842554 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:52.842501 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed949d1_a211_4bc3_8e1f_37dc2eeebefd.slice/crio-efdb803bf4ca91b840481d534b4ab0ae590680e79448b27fbda5c5cb9bcdf66f WatchSource:0}: Error finding container efdb803bf4ca91b840481d534b4ab0ae590680e79448b27fbda5c5cb9bcdf66f: Status 404 returned error can't find the container with id efdb803bf4ca91b840481d534b4ab0ae590680e79448b27fbda5c5cb9bcdf66f Nov 03 20:15:53.136531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:53.133563 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" event={"ID":"bed949d1-a211-4bc3-8e1f-37dc2eeebefd","Type":"ContainerStarted","Data":"efdb803bf4ca91b840481d534b4ab0ae590680e79448b27fbda5c5cb9bcdf66f"} Nov 03 20:15:54.168534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:54.167528 5815 generic.go:356] "Generic (PLEG): container finished" podID="bed949d1-a211-4bc3-8e1f-37dc2eeebefd" containerID="70b1e7a90d6fc2a44496fcca7e608dcdc270725369f6453331c29fbe2998724c" exitCode=0 Nov 03 20:15:54.168534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:54.167702 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" event={"ID":"bed949d1-a211-4bc3-8e1f-37dc2eeebefd","Type":"ContainerDied","Data":"70b1e7a90d6fc2a44496fcca7e608dcdc270725369f6453331c29fbe2998724c"} Nov 03 20:15:55.067535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:55.066600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:55.483474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:55.482613 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:55.667682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:55.663602 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:56.053536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.048844 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:56.121655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.119587 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:56.121655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.119960 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:15:56.507912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.507860 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" event={"ID":"bed949d1-a211-4bc3-8e1f-37dc2eeebefd","Type":"ContainerDied","Data":"efdb803bf4ca91b840481d534b4ab0ae590680e79448b27fbda5c5cb9bcdf66f"} Nov 03 20:15:56.508548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.508492 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdb803bf4ca91b840481d534b4ab0ae590680e79448b27fbda5c5cb9bcdf66f" Nov 03 20:15:56.522581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.513629 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:15:56.758771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.758690 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:56.859992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.859960 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca\") pod \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " Nov 03 20:15:56.860626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.860601 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7\" (UniqueName: \"kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7\") pod \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " Nov 03 20:15:56.860824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.860806 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca\") pod \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\" (UID: \"bed949d1-a211-4bc3-8e1f-37dc2eeebefd\") " Nov 03 20:15:56.862132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.862108 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca" pod "bed949d1-a211-4bc3-8e1f-37dc2eeebefd" (UID: "bed949d1-a211-4bc3-8e1f-37dc2eeebefd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:56.911539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.911480 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca" pod "bed949d1-a211-4bc3-8e1f-37dc2eeebefd" (UID: "bed949d1-a211-4bc3-8e1f-37dc2eeebefd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:56.921769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.921739 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7" pod "bed949d1-a211-4bc3-8e1f-37dc2eeebefd" (UID: "bed949d1-a211-4bc3-8e1f-37dc2eeebefd"). InnerVolumeSpecName "kube-api-access-r9tr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:57.000530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.975879 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9tr7\" (UniqueName: \"kubernetes.io/projected/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-kube-api-access-r9tr7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:57.000530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.975915 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:57.000530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:56.975931 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed949d1-a211-4bc3-8e1f-37dc2eeebefd-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:57.088977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.088878 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.131556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.097584 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.131556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.099363 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:57.296136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.296095 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:15:57.337541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.333269 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.414794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.414695 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 20:15:57.517534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.511587 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.663634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:57.663601 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm" Nov 03 20:15:58.395388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.395358 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg"] Nov 03 20:15:58.395898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.395881 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bed949d1-a211-4bc3-8e1f-37dc2eeebefd" containerName="revoke-keys" Nov 03 20:15:58.395898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.395897 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed949d1-a211-4bc3-8e1f-37dc2eeebefd" containerName="revoke-keys" Nov 03 20:15:58.396037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.395985 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed949d1-a211-4bc3-8e1f-37dc2eeebefd" containerName="revoke-keys" Nov 03 20:15:58.399662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.399639 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.409672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.409649 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg"] Nov 03 20:15:58.410184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.410166 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:15:58.410562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.410545 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:58.476798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.476749 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8xs\" (UniqueName: \"kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.476993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.476895 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.477059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.476992 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.483471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.483446 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm"] Nov 03 20:15:58.489373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.489353 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.495656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.495616 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm"] Nov 03 20:15:58.578048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.578010 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.578453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.578157 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8xs\" (UniqueName: \"kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.578453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.578300 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.578453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.578343 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.578727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.578685 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.578958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.578939 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2r8\" (UniqueName: \"kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.579306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.579284 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.581191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.581170 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.590567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.590545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8xs\" (UniqueName: \"kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs\") pod \"maas-api-revoke-sub-f1abe20f-7b6cg\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.679687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.679603 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.679848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.679760 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.679848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.679812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2r8\" (UniqueName: \"kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.680564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.680526 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.682397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.682374 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.697365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.697342 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2r8\" (UniqueName: \"kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8\") pod \"maas-api-revoke-sub-582fd919-dghcm\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:58.717341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.717303 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:15:58.809135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:58.808709 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:15:59.010919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.010890 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg"] Nov 03 20:15:59.012004 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:59.011980 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c5d4dd_992b_4d96_b86b_cb856d873bd6.slice/crio-3aad9a9c62a372193560eadfac482138634979925f55814be7b64302e626323d WatchSource:0}: Error finding container 3aad9a9c62a372193560eadfac482138634979925f55814be7b64302e626323d: Status 404 returned error can't find the container with id 3aad9a9c62a372193560eadfac482138634979925f55814be7b64302e626323d Nov 03 20:15:59.097815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.097789 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm"] Nov 03 20:15:59.098645 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:15:59.098619 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d194f4e_2b2b_4c54_a7e7_d44c8c4d7f9d.slice/crio-19258d6221193a8bad9cd2258e7acf6b98da5caae4fc22b8f1a6a59ab4b10012 WatchSource:0}: Error finding container 19258d6221193a8bad9cd2258e7acf6b98da5caae4fc22b8f1a6a59ab4b10012: Status 404 returned error can't find the container with id 19258d6221193a8bad9cd2258e7acf6b98da5caae4fc22b8f1a6a59ab4b10012 Nov 03 20:15:59.540637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.540553 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw"] Nov 03 20:15:59.544852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.544825 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.559463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.559437 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw"] Nov 03 20:15:59.591405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591370 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591519 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591564 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591620 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591670 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8k6\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591709 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591756 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591820 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.591877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.591858 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692453 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692521 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8k6\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692564 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692589 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692612 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692687 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692721 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692807 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.692880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692827 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.693114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.692929 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.693372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.693346 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.693465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.693403 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.693680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.693634 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.693979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.693950 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.695599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.695569 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.695702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.695674 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.697583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.697547 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" event={"ID":"a8c5d4dd-992b-4d96-b86b-cb856d873bd6","Type":"ContainerStarted","Data":"17390cb5e887b8695875affb632ff71a5885eb88b5e5035c03edfa10147b476d"} Nov 03 20:15:59.697690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.697604 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" event={"ID":"a8c5d4dd-992b-4d96-b86b-cb856d873bd6","Type":"ContainerStarted","Data":"3aad9a9c62a372193560eadfac482138634979925f55814be7b64302e626323d"} Nov 03 20:15:59.699404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.699377 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" event={"ID":"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d","Type":"ContainerStarted","Data":"cbd909939f5b182738e6a2356d1d88ac4ba53755ebdd7f1113373ee788415961"} Nov 03 20:15:59.699494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.699415 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" event={"ID":"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d","Type":"ContainerStarted","Data":"19258d6221193a8bad9cd2258e7acf6b98da5caae4fc22b8f1a6a59ab4b10012"} Nov 03 20:15:59.704765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.704741 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.705152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.705133 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8k6\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:15:59.724448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.724405 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" podStartSLOduration=1.724394102 podStartE2EDuration="1.724394102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:59.722955111 +0000 UTC m=+869.253300784" watchObservedRunningTime="2026-11-03 20:15:59.724394102 +0000 UTC m=+869.254739776" Nov 03 20:15:59.745768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.745729 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" podStartSLOduration=1.745717668 podStartE2EDuration="1.745717668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:59.741710883 +0000 UTC m=+869.272056618" watchObservedRunningTime="2026-11-03 20:15:59.745717668 +0000 UTC m=+869.276063394" Nov 03 20:15:59.867618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:15:59.867590 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:16:00.179612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.179586 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw"] Nov 03 20:16:00.180812 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:00.180780 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f9fe17_453c_414a_b557_b2bae82e27b2.slice/crio-043eb05d2e50abe38a27ef570126b1e68872fcfa7666cb714e210e12c06a726d WatchSource:0}: Error finding container 043eb05d2e50abe38a27ef570126b1e68872fcfa7666cb714e210e12c06a726d: Status 404 returned error can't find the container with id 043eb05d2e50abe38a27ef570126b1e68872fcfa7666cb714e210e12c06a726d Nov 03 20:16:00.183400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.183366 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:00.183494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.183432 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:00.183494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.183464 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:00.710941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.710890 5815 generic.go:356] "Generic (PLEG): container finished" podID="a8c5d4dd-992b-4d96-b86b-cb856d873bd6" containerID="17390cb5e887b8695875affb632ff71a5885eb88b5e5035c03edfa10147b476d" exitCode=0 Nov 03 20:16:00.711369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.711032 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" event={"ID":"a8c5d4dd-992b-4d96-b86b-cb856d873bd6","Type":"ContainerDied","Data":"17390cb5e887b8695875affb632ff71a5885eb88b5e5035c03edfa10147b476d"} Nov 03 20:16:00.713411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.713384 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" event={"ID":"06f9fe17-453c-414a-b557-b2bae82e27b2","Type":"ContainerStarted","Data":"56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0"} Nov 03 20:16:00.713633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.713614 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" event={"ID":"06f9fe17-453c-414a-b557-b2bae82e27b2","Type":"ContainerStarted","Data":"043eb05d2e50abe38a27ef570126b1e68872fcfa7666cb714e210e12c06a726d"} Nov 03 20:16:00.721423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.719565 5815 generic.go:356] "Generic (PLEG): container finished" podID="3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" containerID="cbd909939f5b182738e6a2356d1d88ac4ba53755ebdd7f1113373ee788415961" exitCode=0 Nov 03 20:16:00.721423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.719680 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" event={"ID":"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d","Type":"ContainerDied","Data":"cbd909939f5b182738e6a2356d1d88ac4ba53755ebdd7f1113373ee788415961"} Nov 03 20:16:00.868716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:00.868620 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:16:01.901534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:01.894621 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:16:01.901534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:01.894681 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:16:01.935920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:01.935894 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:16:01.959204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:01.959021 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podStartSLOduration=2.959002595 podStartE2EDuration="2.959002595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:00.814714629 +0000 UTC m=+870.345060337" watchObservedRunningTime="2026-11-03 20:16:01.959002595 +0000 UTC m=+871.489348268" Nov 03 20:16:02.006882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.006804 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:16:02.031358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.031323 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8\" (UniqueName: \"kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8\") pod \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " Nov 03 20:16:02.031533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.031378 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca\") pod \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " Nov 03 20:16:02.031596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.031539 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca\") pod \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " Nov 03 20:16:02.031669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.031646 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca\") pod \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\" (UID: \"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d\") " Nov 03 20:16:02.031730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.031699 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs\" (UniqueName: \"kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs\") pod \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " Nov 03 20:16:02.031793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.031734 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca\") pod \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\" (UID: \"a8c5d4dd-992b-4d96-b86b-cb856d873bd6\") " Nov 03 20:16:02.033004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.032971 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca" pod "3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" (UID: "3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:02.033145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.033065 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca" pod "a8c5d4dd-992b-4d96-b86b-cb856d873bd6" (UID: "a8c5d4dd-992b-4d96-b86b-cb856d873bd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:02.037123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.037080 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8" pod "3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" (UID: "3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d"). InnerVolumeSpecName "kube-api-access-hg2r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:02.038266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.038223 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca" pod "3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" (UID: "3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:02.038452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.038430 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca" pod "a8c5d4dd-992b-4d96-b86b-cb856d873bd6" (UID: "a8c5d4dd-992b-4d96-b86b-cb856d873bd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:02.039650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.039626 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs" pod "a8c5d4dd-992b-4d96-b86b-cb856d873bd6" (UID: "a8c5d4dd-992b-4d96-b86b-cb856d873bd6"). InnerVolumeSpecName "kube-api-access-2j8xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:02.133699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.133670 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.133699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.133696 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.133699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.133707 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8xs\" (UniqueName: \"kubernetes.io/projected/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-kube-api-access-2j8xs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.133922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.133716 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.133922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.133725 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg2r8\" (UniqueName: \"kubernetes.io/projected/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d-kube-api-access-hg2r8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.133922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.133734 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c5d4dd-992b-4d96-b86b-cb856d873bd6-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:02.736964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.736935 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" Nov 03 20:16:02.737139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.736964 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm" event={"ID":"3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d","Type":"ContainerDied","Data":"19258d6221193a8bad9cd2258e7acf6b98da5caae4fc22b8f1a6a59ab4b10012"} Nov 03 20:16:02.737139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.737011 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19258d6221193a8bad9cd2258e7acf6b98da5caae4fc22b8f1a6a59ab4b10012" Nov 03 20:16:02.741522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.739736 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" Nov 03 20:16:02.741522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.740226 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg" event={"ID":"a8c5d4dd-992b-4d96-b86b-cb856d873bd6","Type":"ContainerDied","Data":"3aad9a9c62a372193560eadfac482138634979925f55814be7b64302e626323d"} Nov 03 20:16:02.741522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.740264 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aad9a9c62a372193560eadfac482138634979925f55814be7b64302e626323d" Nov 03 20:16:02.871588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.868847 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:16:02.871588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:02.868901 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:16:03.877535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:03.872860 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:16:03.877535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:03.872922 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:16:04.869858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.869822 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:16:04.870105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.870076 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:16:04.936700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.936663 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t"] Nov 03 20:16:04.937183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.937170 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" containerName="revoke-keys" Nov 03 20:16:04.937224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.937184 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" containerName="revoke-keys" Nov 03 20:16:04.937224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.937193 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a8c5d4dd-992b-4d96-b86b-cb856d873bd6" containerName="revoke-keys" Nov 03 20:16:04.937224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.937198 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c5d4dd-992b-4d96-b86b-cb856d873bd6" containerName="revoke-keys" Nov 03 20:16:04.937315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.937274 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" containerName="revoke-keys" Nov 03 20:16:04.937315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.937282 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c5d4dd-992b-4d96-b86b-cb856d873bd6" containerName="revoke-keys" Nov 03 20:16:04.940963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.940943 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:04.947030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.946474 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:04.947030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.946663 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:16:04.950833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:04.949975 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t"] Nov 03 20:16:05.096087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.096049 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44f9c\" (UniqueName: \"kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.096245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.096192 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.096284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.096245 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.197733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.197544 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44f9c\" (UniqueName: \"kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.197733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.197645 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.197733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.197686 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.199758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.199658 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.200956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.200934 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.214528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.214480 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44f9c\" (UniqueName: \"kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c\") pod \"maas-api-revoke-sub-a7228736-h8s9t\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.250795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.250775 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:16:05.259270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.259254 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:05.601537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.590022 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t"] Nov 03 20:16:05.801533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.796373 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" event={"ID":"d80a7c99-f35f-4267-ba57-4b5049498ad1","Type":"ContainerStarted","Data":"8d019e5bebcbc1a882f9e65995df5983e05ea0bcbd52eb019daf1a406d7087eb"} Nov 03 20:16:05.876989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.876904 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:16:05.877659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:05.877197 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:16:06.820753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:06.820718 5815 generic.go:356] "Generic (PLEG): container finished" podID="d80a7c99-f35f-4267-ba57-4b5049498ad1" containerID="74a6def0f8b33941b229f7e82ae957a3bfb715793b7e8d8f78e36e9234e9e9ea" exitCode=0 Nov 03 20:16:06.821631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:06.821603 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" event={"ID":"d80a7c99-f35f-4267-ba57-4b5049498ad1","Type":"ContainerDied","Data":"74a6def0f8b33941b229f7e82ae957a3bfb715793b7e8d8f78e36e9234e9e9ea"} Nov 03 20:16:06.869950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:06.869914 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:16:06.870242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:06.870213 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 20:16:08.399879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.399852 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:08.520400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.520315 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c\" (UniqueName: \"kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c\") pod \"d80a7c99-f35f-4267-ba57-4b5049498ad1\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " Nov 03 20:16:08.520604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.520472 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca\") pod \"d80a7c99-f35f-4267-ba57-4b5049498ad1\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " Nov 03 20:16:08.520604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.520530 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca\") pod \"d80a7c99-f35f-4267-ba57-4b5049498ad1\" (UID: \"d80a7c99-f35f-4267-ba57-4b5049498ad1\") " Nov 03 20:16:08.521268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.520832 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca" pod "d80a7c99-f35f-4267-ba57-4b5049498ad1" (UID: "d80a7c99-f35f-4267-ba57-4b5049498ad1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:08.523385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.523359 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c" pod "d80a7c99-f35f-4267-ba57-4b5049498ad1" (UID: "d80a7c99-f35f-4267-ba57-4b5049498ad1"). InnerVolumeSpecName "kube-api-access-44f9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:08.523565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.523362 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca" pod "d80a7c99-f35f-4267-ba57-4b5049498ad1" (UID: "d80a7c99-f35f-4267-ba57-4b5049498ad1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:08.622428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.622391 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44f9c\" (UniqueName: \"kubernetes.io/projected/d80a7c99-f35f-4267-ba57-4b5049498ad1-kube-api-access-44f9c\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:08.622706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.622684 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:08.622821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.622809 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80a7c99-f35f-4267-ba57-4b5049498ad1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:08.682460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.682431 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:16:08.683294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.683262 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:16:08.684198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.684176 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:16:08.871887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.871811 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" Nov 03 20:16:08.871887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.871842 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t" event={"ID":"d80a7c99-f35f-4267-ba57-4b5049498ad1","Type":"ContainerDied","Data":"8d019e5bebcbc1a882f9e65995df5983e05ea0bcbd52eb019daf1a406d7087eb"} Nov 03 20:16:08.872114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:08.871920 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d019e5bebcbc1a882f9e65995df5983e05ea0bcbd52eb019daf1a406d7087eb" Nov 03 20:16:10.127270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.127237 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd"] Nov 03 20:16:10.128023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.128002 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d80a7c99-f35f-4267-ba57-4b5049498ad1" containerName="revoke-keys" Nov 03 20:16:10.128067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.128028 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80a7c99-f35f-4267-ba57-4b5049498ad1" containerName="revoke-keys" Nov 03 20:16:10.128193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.128180 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80a7c99-f35f-4267-ba57-4b5049498ad1" containerName="revoke-keys" Nov 03 20:16:10.132372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.132353 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.137280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.137253 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:10.137430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.137323 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:16:10.140136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.140112 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd"] Nov 03 20:16:10.256410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.256379 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.256613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.256414 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2t57\" (UniqueName: \"kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.256695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.256612 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.369533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.360255 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.369533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.360306 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2t57\" (UniqueName: \"kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.369533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.360374 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.369533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.361160 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.395764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.394668 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.426540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.425594 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2t57\" (UniqueName: \"kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57\") pod \"maas-api-revoke-sub-e6666e35-cpbbd\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.452744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.452706 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:10.766706 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:10.766661 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519543cd_7ad0_42fd_ad76_4db07e5d6782.slice/crio-98bbd2f7f461b810fe4c646eb75a8c42730b59d8d9a4ebaa704fdc7c8b86d68b WatchSource:0}: Error finding container 98bbd2f7f461b810fe4c646eb75a8c42730b59d8d9a4ebaa704fdc7c8b86d68b: Status 404 returned error can't find the container with id 98bbd2f7f461b810fe4c646eb75a8c42730b59d8d9a4ebaa704fdc7c8b86d68b Nov 03 20:16:10.767544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.767486 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd"] Nov 03 20:16:10.894861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:10.894825 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" event={"ID":"519543cd-7ad0-42fd-ad76-4db07e5d6782","Type":"ContainerStarted","Data":"98bbd2f7f461b810fe4c646eb75a8c42730b59d8d9a4ebaa704fdc7c8b86d68b"} Nov 03 20:16:11.223655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.223623 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs"] Nov 03 20:16:11.227480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.227464 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.237635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.237610 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs"] Nov 03 20:16:11.385031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.384961 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.385031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.385003 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.385262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.385148 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dct6\" (UniqueName: \"kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.486617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.486581 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.486818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.486709 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dct6\" (UniqueName: \"kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.486900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.486866 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.487247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.487223 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.489664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.489637 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.499036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.499010 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dct6\" (UniqueName: \"kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6\") pod \"maas-api-revoke-sub-436fd837-rczvs\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.544248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.544216 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:11.828337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.828311 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs"] Nov 03 20:16:11.829351 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:11.829324 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a604c0_02b9_45d6_b30c_718d37d5d0c7.slice/crio-bef8deedf77e9090f07bee2c2f0c83799a2af0db044b7afed89cb44f67663833 WatchSource:0}: Error finding container bef8deedf77e9090f07bee2c2f0c83799a2af0db044b7afed89cb44f67663833: Status 404 returned error can't find the container with id bef8deedf77e9090f07bee2c2f0c83799a2af0db044b7afed89cb44f67663833 Nov 03 20:16:11.906016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.905984 5815 generic.go:356] "Generic (PLEG): container finished" podID="519543cd-7ad0-42fd-ad76-4db07e5d6782" containerID="a017279d7fcc3da36a2fe69f45a3c6975301889f3bd4e2c64cb937e5ebca077f" exitCode=0 Nov 03 20:16:11.906165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.906073 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" event={"ID":"519543cd-7ad0-42fd-ad76-4db07e5d6782","Type":"ContainerDied","Data":"a017279d7fcc3da36a2fe69f45a3c6975301889f3bd4e2c64cb937e5ebca077f"} Nov 03 20:16:11.907520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:11.907481 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" event={"ID":"a1a604c0-02b9-45d6-b30c-718d37d5d0c7","Type":"ContainerStarted","Data":"bef8deedf77e9090f07bee2c2f0c83799a2af0db044b7afed89cb44f67663833"} Nov 03 20:16:12.916835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:12.916796 5815 generic.go:356] "Generic (PLEG): container finished" podID="a1a604c0-02b9-45d6-b30c-718d37d5d0c7" containerID="e4f67bfaaa8dc88f8cfee1b4e389ba6f4e77d4f626b79ae7f935c3aca98ad1a0" exitCode=0 Nov 03 20:16:12.917257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:12.916888 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" event={"ID":"a1a604c0-02b9-45d6-b30c-718d37d5d0c7","Type":"ContainerDied","Data":"e4f67bfaaa8dc88f8cfee1b4e389ba6f4e77d4f626b79ae7f935c3aca98ad1a0"} Nov 03 20:16:13.118477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.118454 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:13.122188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.122163 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca\") pod \"519543cd-7ad0-42fd-ad76-4db07e5d6782\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " Nov 03 20:16:13.122320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.122216 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57\" (UniqueName: \"kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57\") pod \"519543cd-7ad0-42fd-ad76-4db07e5d6782\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " Nov 03 20:16:13.122320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.122294 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca\") pod \"519543cd-7ad0-42fd-ad76-4db07e5d6782\" (UID: \"519543cd-7ad0-42fd-ad76-4db07e5d6782\") " Nov 03 20:16:13.122716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.122690 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca" pod "519543cd-7ad0-42fd-ad76-4db07e5d6782" (UID: "519543cd-7ad0-42fd-ad76-4db07e5d6782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:13.124285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.124257 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca" pod "519543cd-7ad0-42fd-ad76-4db07e5d6782" (UID: "519543cd-7ad0-42fd-ad76-4db07e5d6782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:13.124397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.124367 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57" pod "519543cd-7ad0-42fd-ad76-4db07e5d6782" (UID: "519543cd-7ad0-42fd-ad76-4db07e5d6782"). InnerVolumeSpecName "kube-api-access-w2t57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:13.223568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.223538 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:13.223568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.223567 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519543cd-7ad0-42fd-ad76-4db07e5d6782-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:13.223746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.223577 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2t57\" (UniqueName: \"kubernetes.io/projected/519543cd-7ad0-42fd-ad76-4db07e5d6782-kube-api-access-w2t57\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:13.484861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.484786 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb"] Nov 03 20:16:13.485301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.485287 5815 cpu_manager.go:414] "containerMap: removing container" podUID="519543cd-7ad0-42fd-ad76-4db07e5d6782" containerName="revoke-keys" Nov 03 20:16:13.485353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.485302 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519543cd-7ad0-42fd-ad76-4db07e5d6782" containerName="revoke-keys" Nov 03 20:16:13.485402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.485394 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="519543cd-7ad0-42fd-ad76-4db07e5d6782" containerName="revoke-keys" Nov 03 20:16:13.488703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.488683 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.505238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.504807 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb"] Nov 03 20:16:13.527311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527277 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527327 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527385 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527434 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527606 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527658 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527703 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527787 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckff\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.527936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.527844 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.623426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.623393 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm"] Nov 03 20:16:13.627239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.627218 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.628588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.628561 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.628859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.628833 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.628971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.628906 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.628971 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.628992 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.629034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629028 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.629199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629088 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckff\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629135 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbfl7\" (UniqueName: \"kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.629199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629171 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629258 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629300 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629329 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629357 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629530 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.629946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.629847 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.630057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.630026 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.630174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.630094 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.632100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.632080 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.632341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.632324 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.640919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.640897 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm"] Nov 03 20:16:13.644949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.644928 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.645410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.645390 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckff\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.730012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.729981 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.730163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.730028 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.730163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.730087 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbfl7\" (UniqueName: \"kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.730985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.730953 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.732472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.732435 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.743351 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.743296 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbfl7\" (UniqueName: \"kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7\") pod \"maas-api-revoke-sub-f99718ec-89kdm\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:13.809598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.809568 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:13.862552 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:13.861702 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:16:13.938423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.938382 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" Nov 03 20:16:13.941694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.941594 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd" event={"ID":"519543cd-7ad0-42fd-ad76-4db07e5d6782","Type":"ContainerDied","Data":"98bbd2f7f461b810fe4c646eb75a8c42730b59d8d9a4ebaa704fdc7c8b86d68b"} Nov 03 20:16:13.941852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.941653 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bbd2f7f461b810fe4c646eb75a8c42730b59d8d9a4ebaa704fdc7c8b86d68b" Nov 03 20:16:13.973937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:13.973718 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:14.126978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.124132 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb"] Nov 03 20:16:14.126978 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:14.124547 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1199af3_96ca_4907_9aff_cd92be8c8427.slice/crio-a344140047fc9b80dd4c4dde4347a317c36b9fbb8511e354354aead5409546a6 WatchSource:0}: Error finding container a344140047fc9b80dd4c4dde4347a317c36b9fbb8511e354354aead5409546a6: Status 404 returned error can't find the container with id a344140047fc9b80dd4c4dde4347a317c36b9fbb8511e354354aead5409546a6 Nov 03 20:16:14.126978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.126424 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:14.126978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.126482 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:14.126978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.126551 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:14.254217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.254193 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:14.320238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.320211 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm"] Nov 03 20:16:14.340220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.340198 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca\") pod \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " Nov 03 20:16:14.340350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.340238 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6\" (UniqueName: \"kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6\") pod \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " Nov 03 20:16:14.340350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.340307 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca\") pod \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\" (UID: \"a1a604c0-02b9-45d6-b30c-718d37d5d0c7\") " Nov 03 20:16:14.340805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.340784 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca" pod "a1a604c0-02b9-45d6-b30c-718d37d5d0c7" (UID: "a1a604c0-02b9-45d6-b30c-718d37d5d0c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:14.342589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.342564 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6" pod "a1a604c0-02b9-45d6-b30c-718d37d5d0c7" (UID: "a1a604c0-02b9-45d6-b30c-718d37d5d0c7"). InnerVolumeSpecName "kube-api-access-6dct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:14.342761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.342736 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca" pod "a1a604c0-02b9-45d6-b30c-718d37d5d0c7" (UID: "a1a604c0-02b9-45d6-b30c-718d37d5d0c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:14.446534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.442358 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:14.446534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.442405 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dct6\" (UniqueName: \"kubernetes.io/projected/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-kube-api-access-6dct6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:14.446534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.442421 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1a604c0-02b9-45d6-b30c-718d37d5d0c7-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:14.577631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.577560 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc"] Nov 03 20:16:14.578562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.578474 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a1a604c0-02b9-45d6-b30c-718d37d5d0c7" containerName="revoke-keys" Nov 03 20:16:14.578562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.578499 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a604c0-02b9-45d6-b30c-718d37d5d0c7" containerName="revoke-keys" Nov 03 20:16:14.578737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.578671 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a604c0-02b9-45d6-b30c-718d37d5d0c7" containerName="revoke-keys" Nov 03 20:16:14.586009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.585978 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.605694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.605669 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc"] Nov 03 20:16:14.645379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645314 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645390 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645444 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645548 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645639 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645697 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645730 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7s6\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645850 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.645934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.645889 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.750858 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.750910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.750963 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751014 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751088 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751124 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751149 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7s6\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751224 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751272 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.751924 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.753551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.752180 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.757763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.757726 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.758735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.758705 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.759086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.759016 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.763818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.763758 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.772772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.772690 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.775237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.775153 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7s6\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.780893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.779755 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.908009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.907913 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:14.947193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.947170 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" Nov 03 20:16:14.947995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.947961 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs" event={"ID":"a1a604c0-02b9-45d6-b30c-718d37d5d0c7","Type":"ContainerDied","Data":"bef8deedf77e9090f07bee2c2f0c83799a2af0db044b7afed89cb44f67663833"} Nov 03 20:16:14.948111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.948025 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef8deedf77e9090f07bee2c2f0c83799a2af0db044b7afed89cb44f67663833" Nov 03 20:16:14.953391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.952781 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" event={"ID":"a5258391-537b-490d-bde5-9afd62127221","Type":"ContainerStarted","Data":"8a88496479e0584397b52726a0f76ff60cb51ebf2a1ac255547cbe628780baf1"} Nov 03 20:16:14.953391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.952828 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" event={"ID":"a5258391-537b-490d-bde5-9afd62127221","Type":"ContainerStarted","Data":"6c0f08f6f5767b189b44f79006f513f06c234340d4b60cbb0f1f0cf5decf233d"} Nov 03 20:16:14.955823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.955054 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" event={"ID":"b1199af3-96ca-4907-9aff-cd92be8c8427","Type":"ContainerStarted","Data":"1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9"} Nov 03 20:16:14.955823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.955090 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" event={"ID":"b1199af3-96ca-4907-9aff-cd92be8c8427","Type":"ContainerStarted","Data":"a344140047fc9b80dd4c4dde4347a317c36b9fbb8511e354354aead5409546a6"} Nov 03 20:16:15.011533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:14.979414 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" podStartSLOduration=1.9793938469999999 podStartE2EDuration="1.979393847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:14.97882609 +0000 UTC m=+884.509171759" watchObservedRunningTime="2026-11-03 20:16:14.979393847 +0000 UTC m=+884.509739524" Nov 03 20:16:15.043366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.043316 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podStartSLOduration=2.043301028 podStartE2EDuration="2.043301028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:15.023028272 +0000 UTC m=+884.553373949" watchObservedRunningTime="2026-11-03 20:16:15.043301028 +0000 UTC m=+884.573646699" Nov 03 20:16:15.315329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.315299 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc"] Nov 03 20:16:15.321835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.321798 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:15.321947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.321870 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:15.321947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.321903 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:15.823533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.823130 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:15.823533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.823229 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:16:15.823533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.823264 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:16:15.999220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.987583 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5258391-537b-490d-bde5-9afd62127221" containerID="8a88496479e0584397b52726a0f76ff60cb51ebf2a1ac255547cbe628780baf1" exitCode=0 Nov 03 20:16:15.999220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:15.987715 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" event={"ID":"a5258391-537b-490d-bde5-9afd62127221","Type":"ContainerDied","Data":"8a88496479e0584397b52726a0f76ff60cb51ebf2a1ac255547cbe628780baf1"} Nov 03 20:16:16.009767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.009035 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" event={"ID":"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff","Type":"ContainerStarted","Data":"692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473"} Nov 03 20:16:16.009767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.009093 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" event={"ID":"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff","Type":"ContainerStarted","Data":"815b7b099c99ab1e3c03ce0012dad9ca68d145e92028c2b1ae7f96b502e205fd"} Nov 03 20:16:16.054519 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.054447 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podStartSLOduration=2.054429226 podStartE2EDuration="2.054429226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:16.053910529 +0000 UTC m=+885.584256203" watchObservedRunningTime="2026-11-03 20:16:16.054429226 +0000 UTC m=+885.584774905" Nov 03 20:16:16.812573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.811829 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:16:16.812573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.811894 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:16:16.910380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.910347 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:16.910754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.910729 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:16:16.910918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:16.910893 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 20:16:17.330255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.330221 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:17.428702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.428611 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca\") pod \"a5258391-537b-490d-bde5-9afd62127221\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " Nov 03 20:16:17.428885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.428742 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca\") pod \"a5258391-537b-490d-bde5-9afd62127221\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " Nov 03 20:16:17.428885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.428862 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7\" (UniqueName: \"kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7\") pod \"a5258391-537b-490d-bde5-9afd62127221\" (UID: \"a5258391-537b-490d-bde5-9afd62127221\") " Nov 03 20:16:17.429446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.429414 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca" pod "a5258391-537b-490d-bde5-9afd62127221" (UID: "a5258391-537b-490d-bde5-9afd62127221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:17.431488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.431446 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7" pod "a5258391-537b-490d-bde5-9afd62127221" (UID: "a5258391-537b-490d-bde5-9afd62127221"). InnerVolumeSpecName "kube-api-access-xbfl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:17.431755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.431734 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca" pod "a5258391-537b-490d-bde5-9afd62127221" (UID: "a5258391-537b-490d-bde5-9afd62127221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:17.530924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.530887 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbfl7\" (UniqueName: \"kubernetes.io/projected/a5258391-537b-490d-bde5-9afd62127221-kube-api-access-xbfl7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:17.530924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.530917 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5258391-537b-490d-bde5-9afd62127221-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:17.530924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.530927 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5258391-537b-490d-bde5-9afd62127221-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:17.819535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.813911 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:16:17.819535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.813976 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:16:17.920054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.920017 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:16:17.920346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:17.920299 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 20:16:18.051265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.051229 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" event={"ID":"a5258391-537b-490d-bde5-9afd62127221","Type":"ContainerDied","Data":"6c0f08f6f5767b189b44f79006f513f06c234340d4b60cbb0f1f0cf5decf233d"} Nov 03 20:16:18.051535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.051246 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm" Nov 03 20:16:18.051659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.051643 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0f08f6f5767b189b44f79006f513f06c234340d4b60cbb0f1f0cf5decf233d" Nov 03 20:16:18.425971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.422838 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr"] Nov 03 20:16:18.425971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.424926 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a5258391-537b-490d-bde5-9afd62127221" containerName="revoke-keys" Nov 03 20:16:18.425971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.424950 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5258391-537b-490d-bde5-9afd62127221" containerName="revoke-keys" Nov 03 20:16:18.425971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.425122 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5258391-537b-490d-bde5-9afd62127221" containerName="revoke-keys" Nov 03 20:16:18.429876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.429853 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.439837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.439812 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:18.440087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.440071 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:16:18.447558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.446335 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr"] Nov 03 20:16:18.549307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.549275 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpv4l\" (UniqueName: \"kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.549483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.549340 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.549483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.549371 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.650678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.650644 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.650929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.650911 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.651262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.651242 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpv4l\" (UniqueName: \"kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.651598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.651573 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.659169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.659146 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.663923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.663897 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpv4l\" (UniqueName: \"kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l\") pod \"maas-api-revoke-sub-cd72ef1e-lrlrr\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.749858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.749826 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:18.810947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.810909 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:16:18.811196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.811170 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:16:18.913545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.910930 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:16:18.913545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:18.910990 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 20:16:19.147583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:19.147554 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr"] Nov 03 20:16:19.812530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:19.811718 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:16:19.812530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:19.811780 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:16:19.913533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:19.911891 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:16:19.913533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:19.911949 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 20:16:20.162165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:20.162076 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" event={"ID":"40f1b536-89ed-44db-95b9-0bb53ae16b51","Type":"ContainerStarted","Data":"ba5b70171f804a3382f3593813f4e6f678d794ebee63d7cb14c37250cf701cba"} Nov 03 20:16:20.162165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:20.162159 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" event={"ID":"40f1b536-89ed-44db-95b9-0bb53ae16b51","Type":"ContainerStarted","Data":"a645fcfce1af4b9761af92682010e16e717d4c3577f8ed1c1689e77aa89998fa"} Nov 03 20:16:20.810265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:20.810226 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:16:20.810425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:20.810300 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 20:16:20.909162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:20.909121 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:16:20.909597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:20.909187 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 20:16:21.205613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:21.200012 5815 generic.go:356] "Generic (PLEG): container finished" podID="40f1b536-89ed-44db-95b9-0bb53ae16b51" containerID="ba5b70171f804a3382f3593813f4e6f678d794ebee63d7cb14c37250cf701cba" exitCode=0 Nov 03 20:16:21.205613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:21.200077 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" event={"ID":"40f1b536-89ed-44db-95b9-0bb53ae16b51","Type":"ContainerDied","Data":"ba5b70171f804a3382f3593813f4e6f678d794ebee63d7cb14c37250cf701cba"} Nov 03 20:16:21.908812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:21.908727 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:16:21.908812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:21.908797 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 20:16:22.192449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.183086 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:22.198068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.198045 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:22.219936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.219873 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:16:22.545559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.543902 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88"] Nov 03 20:16:22.550148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.550113 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.557028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.557003 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88"] Nov 03 20:16:22.599427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.599335 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:22.760819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.760785 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca\") pod \"40f1b536-89ed-44db-95b9-0bb53ae16b51\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " Nov 03 20:16:22.777825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.777704 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca\") pod \"40f1b536-89ed-44db-95b9-0bb53ae16b51\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " Nov 03 20:16:22.778368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.778352 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l\" (UniqueName: \"kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l\") pod \"40f1b536-89ed-44db-95b9-0bb53ae16b51\" (UID: \"40f1b536-89ed-44db-95b9-0bb53ae16b51\") " Nov 03 20:16:22.778874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.778852 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca" pod "40f1b536-89ed-44db-95b9-0bb53ae16b51" (UID: "40f1b536-89ed-44db-95b9-0bb53ae16b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:22.785622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.785596 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.785920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.785904 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf92q\" (UniqueName: \"kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.816195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.816114 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.816481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.816467 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:22.851391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.840668 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l" pod "40f1b536-89ed-44db-95b9-0bb53ae16b51" (UID: "40f1b536-89ed-44db-95b9-0bb53ae16b51"). InnerVolumeSpecName "kube-api-access-zpv4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:22.854380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.854340 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca" pod "40f1b536-89ed-44db-95b9-0bb53ae16b51" (UID: "40f1b536-89ed-44db-95b9-0bb53ae16b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:22.917474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.917438 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.917670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.917542 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.917670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.917599 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf92q\" (UniqueName: \"kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.917812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.917797 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f1b536-89ed-44db-95b9-0bb53ae16b51-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:22.917886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.917819 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpv4l\" (UniqueName: \"kubernetes.io/projected/40f1b536-89ed-44db-95b9-0bb53ae16b51-kube-api-access-zpv4l\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:22.918362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.918291 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.920255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.920230 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:22.932166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:22.931674 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf92q\" (UniqueName: \"kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q\") pod \"maas-api-revoke-sub-7c214eae-vfr88\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:23.178577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.176704 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:23.260411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.260369 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" Nov 03 20:16:23.262492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.262458 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr" event={"ID":"40f1b536-89ed-44db-95b9-0bb53ae16b51","Type":"ContainerDied","Data":"a645fcfce1af4b9761af92682010e16e717d4c3577f8ed1c1689e77aa89998fa"} Nov 03 20:16:23.262635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.262533 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a645fcfce1af4b9761af92682010e16e717d4c3577f8ed1c1689e77aa89998fa" Nov 03 20:16:23.367922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.367896 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:23.368856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.368780 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:23.369634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.369613 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 20:16:23.503953 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:23.503906 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5002cd39_5bd6_4d7a_929c_6787ca6c5754.slice/crio-4f599eb860195dde761105e97a967eab5e24afb4bec7e24c31a616b9a9e8804a WatchSource:0}: Error finding container 4f599eb860195dde761105e97a967eab5e24afb4bec7e24c31a616b9a9e8804a: Status 404 returned error can't find the container with id 4f599eb860195dde761105e97a967eab5e24afb4bec7e24c31a616b9a9e8804a Nov 03 20:16:23.507644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.507623 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88"] Nov 03 20:16:23.979631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.979603 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9"] Nov 03 20:16:23.980116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.980104 5815 cpu_manager.go:414] "containerMap: removing container" podUID="40f1b536-89ed-44db-95b9-0bb53ae16b51" containerName="revoke-keys" Nov 03 20:16:23.980159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.980117 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f1b536-89ed-44db-95b9-0bb53ae16b51" containerName="revoke-keys" Nov 03 20:16:23.980204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.980195 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f1b536-89ed-44db-95b9-0bb53ae16b51" containerName="revoke-keys" Nov 03 20:16:23.983652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.983636 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:23.994047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:23.994024 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9"] Nov 03 20:16:24.135588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.135261 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbr2c\" (UniqueName: \"kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.135588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.135313 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.135588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.135369 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.236834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.236800 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbr2c\" (UniqueName: \"kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.236834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.236837 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.237042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.236871 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.237583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.237564 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.239547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.239524 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.249410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.249379 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbr2c\" (UniqueName: \"kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c\") pod \"maas-api-revoke-sub-a775fabb-krxv9\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.271767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.271728 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" event={"ID":"5002cd39-5bd6-4d7a-929c-6787ca6c5754","Type":"ContainerStarted","Data":"709e2d9fa36e3d858a5c7f96d31d25164cc4d8fa0d57cd5f3aca2e0b6878eef7"} Nov 03 20:16:24.272178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.271826 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" event={"ID":"5002cd39-5bd6-4d7a-929c-6787ca6c5754","Type":"ContainerStarted","Data":"4f599eb860195dde761105e97a967eab5e24afb4bec7e24c31a616b9a9e8804a"} Nov 03 20:16:24.293121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.293072 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" podStartSLOduration=2.293055496 podStartE2EDuration="2.293055496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:24.289709191 +0000 UTC m=+893.820054866" watchObservedRunningTime="2026-11-03 20:16:24.293055496 +0000 UTC m=+893.823401169" Nov 03 20:16:24.303540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.303497 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:24.589925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.589887 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9"] Nov 03 20:16:24.590416 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:24.590375 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice/crio-02774fca42e98567c46ab592d6baa959d589429b6512e00e60056e88eced1952 WatchSource:0}: Error finding container 02774fca42e98567c46ab592d6baa959d589429b6512e00e60056e88eced1952: Status 404 returned error can't find the container with id 02774fca42e98567c46ab592d6baa959d589429b6512e00e60056e88eced1952 Nov 03 20:16:24.764018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.763990 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8"] Nov 03 20:16:24.770008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.769980 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.801015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.800993 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8"] Nov 03 20:16:24.836771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.836739 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd"] Nov 03 20:16:24.843807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.842061 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.856564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.856114 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd"] Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947207 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947276 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947307 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947339 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947377 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947402 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947450 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947559 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947602 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947626 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947696 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8qb\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947764 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947795 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947817 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947842 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2xmh\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947874 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947897 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:24.949550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:24.947943 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049082 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049138 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.049292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049168 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049298 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049354 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.049431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049385 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.049596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049433 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.049596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049540 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049579 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049605 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.049760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049612 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.049760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049656 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049614 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.049760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049720 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8qb\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.050015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049827 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.050015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049888 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2xmh\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049955 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.050015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049977 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.050047 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.050344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.050272 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.050344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.050291 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.049954 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.050594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.050543 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.050617 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.050844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.050742 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.051057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.051035 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.052241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.052222 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.052715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.052693 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.052786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.052702 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.052976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.052957 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.067393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.067176 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.067393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.067305 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2xmh\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.067393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.067380 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8qb\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.067642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.067577 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.095801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.095739 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:25.170794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.170464 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:25.287800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.284960 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" event={"ID":"4d7bbbae-6a84-49a8-a59b-771e82999294","Type":"ContainerStarted","Data":"48c5004cd47e1780d5d3f8d7b6c21355084827e9a3304c36109fc07d443d618d"} Nov 03 20:16:25.287800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.285031 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" event={"ID":"4d7bbbae-6a84-49a8-a59b-771e82999294","Type":"ContainerStarted","Data":"02774fca42e98567c46ab592d6baa959d589429b6512e00e60056e88eced1952"} Nov 03 20:16:25.291661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.289061 5815 generic.go:356] "Generic (PLEG): container finished" podID="5002cd39-5bd6-4d7a-929c-6787ca6c5754" containerID="709e2d9fa36e3d858a5c7f96d31d25164cc4d8fa0d57cd5f3aca2e0b6878eef7" exitCode=0 Nov 03 20:16:25.291661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.290330 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" event={"ID":"5002cd39-5bd6-4d7a-929c-6787ca6c5754","Type":"ContainerDied","Data":"709e2d9fa36e3d858a5c7f96d31d25164cc4d8fa0d57cd5f3aca2e0b6878eef7"} Nov 03 20:16:25.425370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.424474 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" podStartSLOduration=2.424449631 podStartE2EDuration="2.424449631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:25.309096666 +0000 UTC m=+894.839442351" watchObservedRunningTime="2026-11-03 20:16:25.424449631 +0000 UTC m=+894.954795304" Nov 03 20:16:25.453454 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:25.453319 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c676a88_ab43_48e5_ab69_8849f20ca3f7.slice/crio-f5bdab8c206f1ebdf863b952c0908d4452f2a69704388b3b868b1f913b389b4d WatchSource:0}: Error finding container f5bdab8c206f1ebdf863b952c0908d4452f2a69704388b3b868b1f913b389b4d: Status 404 returned error can't find the container with id f5bdab8c206f1ebdf863b952c0908d4452f2a69704388b3b868b1f913b389b4d Nov 03 20:16:25.456768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.456079 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:25.456768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.456160 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:25.456768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.456200 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:25.465758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.465583 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8"] Nov 03 20:16:25.494090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.494060 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd"] Nov 03 20:16:25.514283 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:25.514251 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dae19a_562f_4e3e_9136_81672c805be1.slice/crio-7d5b1ddc6147297f7430446f8149b55b3dd8ce60a8e9b6ea4c135993a3cf6d9f WatchSource:0}: Error finding container 7d5b1ddc6147297f7430446f8149b55b3dd8ce60a8e9b6ea4c135993a3cf6d9f: Status 404 returned error can't find the container with id 7d5b1ddc6147297f7430446f8149b55b3dd8ce60a8e9b6ea4c135993a3cf6d9f Nov 03 20:16:25.518005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.517933 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:25.518005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.518010 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:25.518005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:25.518042 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:16:26.368753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.367671 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" event={"ID":"1c676a88-ab43-48e5-ab69-8849f20ca3f7","Type":"ContainerStarted","Data":"c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470"} Nov 03 20:16:26.368753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.367742 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" event={"ID":"1c676a88-ab43-48e5-ab69-8849f20ca3f7","Type":"ContainerStarted","Data":"f5bdab8c206f1ebdf863b952c0908d4452f2a69704388b3b868b1f913b389b4d"} Nov 03 20:16:26.409929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.409870 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podStartSLOduration=2.409851194 podStartE2EDuration="2.409851194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:26.409143224 +0000 UTC m=+895.939488896" watchObservedRunningTime="2026-11-03 20:16:26.409851194 +0000 UTC m=+895.940196866" Nov 03 20:16:26.426808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.425551 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" event={"ID":"79dae19a-562f-4e3e-9136-81672c805be1","Type":"ContainerStarted","Data":"9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f"} Nov 03 20:16:26.426808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.425626 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" event={"ID":"79dae19a-562f-4e3e-9136-81672c805be1","Type":"ContainerStarted","Data":"7d5b1ddc6147297f7430446f8149b55b3dd8ce60a8e9b6ea4c135993a3cf6d9f"} Nov 03 20:16:26.443657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.429941 5815 generic.go:356] "Generic (PLEG): container finished" podID="4d7bbbae-6a84-49a8-a59b-771e82999294" containerID="48c5004cd47e1780d5d3f8d7b6c21355084827e9a3304c36109fc07d443d618d" exitCode=0 Nov 03 20:16:26.443657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.430231 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" event={"ID":"4d7bbbae-6a84-49a8-a59b-771e82999294","Type":"ContainerDied","Data":"48c5004cd47e1780d5d3f8d7b6c21355084827e9a3304c36109fc07d443d618d"} Nov 03 20:16:26.504535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:26.494840 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podStartSLOduration=2.494818687 podStartE2EDuration="2.494818687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:26.472974672 +0000 UTC m=+896.003320346" watchObservedRunningTime="2026-11-03 20:16:26.494818687 +0000 UTC m=+896.025164362" Nov 03 20:16:27.116533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.114294 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:27.116533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.114360 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:27.170531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.126336 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:27.174532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.172385 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:27.174532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.172525 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:27.174532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.172575 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:27.238534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.237625 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 20:16:27.302532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.301707 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.394371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.393598 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.695352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:27.694602 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.073533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.065693 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:28.073533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.073032 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.100951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.100915 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:28.101222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.101192 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:28.136412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.136371 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca\") pod \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " Nov 03 20:16:28.159905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.159769 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca" pod "5002cd39-5bd6-4d7a-929c-6787ca6c5754" (UID: "5002cd39-5bd6-4d7a-929c-6787ca6c5754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:28.160777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.160316 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q\" (UniqueName: \"kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q\") pod \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " Nov 03 20:16:28.161020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.161001 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca\") pod \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\" (UID: \"5002cd39-5bd6-4d7a-929c-6787ca6c5754\") " Nov 03 20:16:28.171757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.171699 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.196573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.196536 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q" pod "5002cd39-5bd6-4d7a-929c-6787ca6c5754" (UID: "5002cd39-5bd6-4d7a-929c-6787ca6c5754"). InnerVolumeSpecName "kube-api-access-vf92q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:28.215564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.205718 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:28.215564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.205779 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:28.219993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.219956 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca" pod "5002cd39-5bd6-4d7a-929c-6787ca6c5754" (UID: "5002cd39-5bd6-4d7a-929c-6787ca6c5754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:28.277531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.274435 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf92q\" (UniqueName: \"kubernetes.io/projected/5002cd39-5bd6-4d7a-929c-6787ca6c5754-kube-api-access-vf92q\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.277531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.274475 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5002cd39-5bd6-4d7a-929c-6787ca6c5754-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.398640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.332651 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.398640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.333423 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:16:28.659639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.659596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.723535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.672601 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:28.734314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.704352 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:16:28.958631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:28.921399 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:16:29.077284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.076931 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:16:29.077466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.077295 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:16:29.099037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.098598 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:29.099037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.098652 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:29.206611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.196627 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:29.206611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.196695 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:29.239935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.239569 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" Nov 03 20:16:29.240205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.240177 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88" event={"ID":"5002cd39-5bd6-4d7a-929c-6787ca6c5754","Type":"ContainerDied","Data":"4f599eb860195dde761105e97a967eab5e24afb4bec7e24c31a616b9a9e8804a"} Nov 03 20:16:29.240396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.240379 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f599eb860195dde761105e97a967eab5e24afb4bec7e24c31a616b9a9e8804a" Nov 03 20:16:29.340883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.340836 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.668922 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g"] Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.669783 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5002cd39-5bd6-4d7a-929c-6787ca6c5754" containerName="revoke-keys" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.669804 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5002cd39-5bd6-4d7a-929c-6787ca6c5754" containerName="revoke-keys" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.669975 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5002cd39-5bd6-4d7a-929c-6787ca6c5754" containerName="revoke-keys" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.681935 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.693696 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.697705 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:16:29.738160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.708460 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g"] Nov 03 20:16:29.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.780248 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:16:29.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.782605 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:29.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.788659 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:16:29.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.797809 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ssdx\" (UniqueName: \"kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.797928 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.797988 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.953532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.941066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.953532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.941267 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ssdx\" (UniqueName: \"kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.953532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.941334 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:29.959331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:29.959299 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:30.251595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.121875 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:16:30.251595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.175792 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:30.251595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.223464 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:30.251595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.192210 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ssdx\" (UniqueName: \"kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:30.251595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.181841 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:30.251595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.223632 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:30.257544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.256692 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19b3dcb6-8g25g\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:30.266630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.259037 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:30.341063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.341022 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:30.390559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.364652 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.428596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.424922 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.481822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.481774 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca\") pod \"4d7bbbae-6a84-49a8-a59b-771e82999294\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " Nov 03 20:16:30.482207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.482183 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c\" (UniqueName: \"kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c\") pod \"4d7bbbae-6a84-49a8-a59b-771e82999294\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " Nov 03 20:16:30.482402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.482385 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca\") pod \"4d7bbbae-6a84-49a8-a59b-771e82999294\" (UID: \"4d7bbbae-6a84-49a8-a59b-771e82999294\") " Nov 03 20:16:30.490453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.490409 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca" pod "4d7bbbae-6a84-49a8-a59b-771e82999294" (UID: "4d7bbbae-6a84-49a8-a59b-771e82999294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:30.576883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.576835 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c" pod "4d7bbbae-6a84-49a8-a59b-771e82999294" (UID: "4d7bbbae-6a84-49a8-a59b-771e82999294"). InnerVolumeSpecName "kube-api-access-bbr2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:30.577169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.577145 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca" pod "4d7bbbae-6a84-49a8-a59b-771e82999294" (UID: "4d7bbbae-6a84-49a8-a59b-771e82999294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:30.591247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.591210 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" event={"ID":"4d7bbbae-6a84-49a8-a59b-771e82999294","Type":"ContainerDied","Data":"02774fca42e98567c46ab592d6baa959d589429b6512e00e60056e88eced1952"} Nov 03 20:16:30.591487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.591465 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02774fca42e98567c46ab592d6baa959d589429b6512e00e60056e88eced1952" Nov 03 20:16:30.591734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.591719 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9" Nov 03 20:16:30.890565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.795817 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:30.922404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.811802 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbr2c\" (UniqueName: \"kubernetes.io/projected/4d7bbbae-6a84-49a8-a59b-771e82999294-kube-api-access-bbr2c\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.984397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.980693 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.984397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.980726 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d7bbbae-6a84-49a8-a59b-771e82999294-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.984397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.922678 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:16:30.984397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:30.921943 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:16:31.375969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.375847 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.940528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.939600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:31.943616 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:31.942709 5815 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.35s" Nov 03 20:16:31.964749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.946847 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:31.964749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.946899 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:31.964749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.946973 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:31.964749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.946997 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:31.974542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.971939 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:16:31.997553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.995099 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.997553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:31.995441 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:16:32.115960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.115925 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:32.205449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.205356 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:32.455588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.451913 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:32.459623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.459583 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:32.961212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.961148 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.994535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.977837 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:16:32.994737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.977914 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:33.024551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:32.995960 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.024551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.019608 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:33.024551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.019673 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.024927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.024890 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:16:33.206013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.185098 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:33.206013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.185158 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:33.206013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.185232 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:33.206013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.185256 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:33.427534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.423372 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g"] Nov 03 20:16:33.810005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.809919 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:16:33.892117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:33.891897 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:16:33.967537 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:33.957188 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:34.041554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.033748 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:34.041554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.034454 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:16:34.041554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.034804 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:16:34.057550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.054391 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:16:34.057550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.054550 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:16:34.057550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.054601 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 20:16:34.152585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.148808 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:16:34.162346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.162314 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:34.164726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.164697 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:34.181479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.181071 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:34.181823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.181794 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:34.182573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.182555 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:16:34.233175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.233148 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:16:34.641539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.637428 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" event={"ID":"e782f559-d001-4cf3-8361-5bc4ba3fe455","Type":"ContainerStarted","Data":"ad70b2c870fc306947f8ce7757385cf95f3bc382b0a9e52354c56a8f989bf548"} Nov 03 20:16:34.641856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.641830 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" event={"ID":"e782f559-d001-4cf3-8361-5bc4ba3fe455","Type":"ContainerStarted","Data":"01119fd0eb0f813572716c3af90dbf2df7668461e14f0087184551369d82c044"} Nov 03 20:16:34.690581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.684991 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:16:34.701264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:34.701204 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" podStartSLOduration=5.7011837 podStartE2EDuration="5.7011837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:34.675899301 +0000 UTC m=+904.206244973" watchObservedRunningTime="2026-11-03 20:16:34.7011837 +0000 UTC m=+904.231529373" Nov 03 20:16:35.038535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.036725 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.046372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.046346 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:16:35.053530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.049149 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:16:35.083417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.080045 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.105102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.096810 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:35.105102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.096856 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:35.174224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.172867 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:35.174224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.172924 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:35.751699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.751666 5815 generic.go:356] "Generic (PLEG): container finished" podID="e782f559-d001-4cf3-8361-5bc4ba3fe455" containerID="ad70b2c870fc306947f8ce7757385cf95f3bc382b0a9e52354c56a8f989bf548" exitCode=0 Nov 03 20:16:35.752648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.752300 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" event={"ID":"e782f559-d001-4cf3-8361-5bc4ba3fe455","Type":"ContainerDied","Data":"ad70b2c870fc306947f8ce7757385cf95f3bc382b0a9e52354c56a8f989bf548"} Nov 03 20:16:35.756572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.756549 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:16:35.757869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:35.757852 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:16:36.100393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:36.100305 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:36.101025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:36.100988 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:36.172558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:36.170870 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:36.172558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:36.170932 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:37.110770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.101634 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:37.110770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.101692 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:37.195541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.172892 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:37.195541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.172946 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:37.226612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.226581 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:37.244125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.244097 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca\") pod \"e782f559-d001-4cf3-8361-5bc4ba3fe455\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " Nov 03 20:16:37.244382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.244366 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca\") pod \"e782f559-d001-4cf3-8361-5bc4ba3fe455\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " Nov 03 20:16:37.244632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.244617 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx\" (UniqueName: \"kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx\") pod \"e782f559-d001-4cf3-8361-5bc4ba3fe455\" (UID: \"e782f559-d001-4cf3-8361-5bc4ba3fe455\") " Nov 03 20:16:37.245447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.245188 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca" pod "e782f559-d001-4cf3-8361-5bc4ba3fe455" (UID: "e782f559-d001-4cf3-8361-5bc4ba3fe455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:37.245971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.245893 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:37.247444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.247426 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx" pod "e782f559-d001-4cf3-8361-5bc4ba3fe455" (UID: "e782f559-d001-4cf3-8361-5bc4ba3fe455"). InnerVolumeSpecName "kube-api-access-2ssdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:37.248482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.248460 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca" pod "e782f559-d001-4cf3-8361-5bc4ba3fe455" (UID: "e782f559-d001-4cf3-8361-5bc4ba3fe455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:37.347542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.347474 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ssdx\" (UniqueName: \"kubernetes.io/projected/e782f559-d001-4cf3-8361-5bc4ba3fe455-kube-api-access-2ssdx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:37.347542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.347536 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e782f559-d001-4cf3-8361-5bc4ba3fe455-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:37.822813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.822768 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" event={"ID":"e782f559-d001-4cf3-8361-5bc4ba3fe455","Type":"ContainerDied","Data":"01119fd0eb0f813572716c3af90dbf2df7668461e14f0087184551369d82c044"} Nov 03 20:16:37.823181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.823165 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01119fd0eb0f813572716c3af90dbf2df7668461e14f0087184551369d82c044" Nov 03 20:16:37.823271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.823124 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g" Nov 03 20:16:37.831143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.831036 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6"] Nov 03 20:16:37.832163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.832134 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4d7bbbae-6a84-49a8-a59b-771e82999294" containerName="revoke-keys" Nov 03 20:16:37.832326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.832313 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d7bbbae-6a84-49a8-a59b-771e82999294" containerName="revoke-keys" Nov 03 20:16:37.832414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.832405 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e782f559-d001-4cf3-8361-5bc4ba3fe455" containerName="revoke-keys" Nov 03 20:16:37.832489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.832479 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e782f559-d001-4cf3-8361-5bc4ba3fe455" containerName="revoke-keys" Nov 03 20:16:37.832807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.832788 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d7bbbae-6a84-49a8-a59b-771e82999294" containerName="revoke-keys" Nov 03 20:16:37.832927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.832910 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e782f559-d001-4cf3-8361-5bc4ba3fe455" containerName="revoke-keys" Nov 03 20:16:37.916860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.916831 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:37.944961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.944927 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:16:38.006479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.916721 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6"] Nov 03 20:16:38.006690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.946467 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:38.006762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:37.987427 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.006821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.006798 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.007003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.006980 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr8l5\" (UniqueName: \"kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.102590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.097560 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:16:38.102590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.097618 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:16:38.108567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.108528 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr8l5\" (UniqueName: \"kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.108893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.108870 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.109066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.109047 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.110394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.110304 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.112408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.112387 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.135537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.135104 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr8l5\" (UniqueName: \"kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5\") pod \"maas-api-revoke-sub-6d195641-5gml6\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.176814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.176778 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:38.177089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.177049 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:38.342732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.342697 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:38.651619 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:38.651586 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c92839_cf49_48b2_a24d_c984020d8393.slice/crio-3f98581098b6b3907cea33a809a56e27ba91dc4dd7430b947da45c5901cde2b4 WatchSource:0}: Error finding container 3f98581098b6b3907cea33a809a56e27ba91dc4dd7430b947da45c5901cde2b4: Status 404 returned error can't find the container with id 3f98581098b6b3907cea33a809a56e27ba91dc4dd7430b947da45c5901cde2b4 Nov 03 20:16:38.652406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.652378 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6"] Nov 03 20:16:38.835055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:38.834942 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" event={"ID":"b2c92839-cf49-48b2-a24d-c984020d8393","Type":"ContainerStarted","Data":"3f98581098b6b3907cea33a809a56e27ba91dc4dd7430b947da45c5901cde2b4"} Nov 03 20:16:39.171538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:39.171483 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 20:16:39.172013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:39.171570 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 20:16:39.413619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:39.413437 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:39.414282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:39.414259 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:39.415280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:39.415258 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:16:39.848898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:39.848855 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" event={"ID":"b2c92839-cf49-48b2-a24d-c984020d8393","Type":"ContainerStarted","Data":"dff89ce8d693b6a6aad22c3176133fc3b1f71a66ab8555b166ca1035392a04f3"} Nov 03 20:16:40.172207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:40.172117 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:40.175430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:40.175406 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:40.202456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:40.202409 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" podStartSLOduration=3.202396326 podStartE2EDuration="3.202396326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:39.870822133 +0000 UTC m=+909.401167806" watchObservedRunningTime="2026-11-03 20:16:40.202396326 +0000 UTC m=+909.732741997" Nov 03 20:16:40.859063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:40.859033 5815 generic.go:356] "Generic (PLEG): container finished" podID="b2c92839-cf49-48b2-a24d-c984020d8393" containerID="dff89ce8d693b6a6aad22c3176133fc3b1f71a66ab8555b166ca1035392a04f3" exitCode=0 Nov 03 20:16:40.859308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:40.859120 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" event={"ID":"b2c92839-cf49-48b2-a24d-c984020d8393","Type":"ContainerDied","Data":"dff89ce8d693b6a6aad22c3176133fc3b1f71a66ab8555b166ca1035392a04f3"} Nov 03 20:16:40.860333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:40.860315 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:16:42.070452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.070428 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:42.167047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.166958 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5\" (UniqueName: \"kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5\") pod \"b2c92839-cf49-48b2-a24d-c984020d8393\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " Nov 03 20:16:42.167216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.167103 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca\") pod \"b2c92839-cf49-48b2-a24d-c984020d8393\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " Nov 03 20:16:42.167216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.167142 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca\") pod \"b2c92839-cf49-48b2-a24d-c984020d8393\" (UID: \"b2c92839-cf49-48b2-a24d-c984020d8393\") " Nov 03 20:16:42.167487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.167459 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca" pod "b2c92839-cf49-48b2-a24d-c984020d8393" (UID: "b2c92839-cf49-48b2-a24d-c984020d8393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:42.169540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.169471 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca" pod "b2c92839-cf49-48b2-a24d-c984020d8393" (UID: "b2c92839-cf49-48b2-a24d-c984020d8393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:42.169692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.169578 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5" pod "b2c92839-cf49-48b2-a24d-c984020d8393" (UID: "b2c92839-cf49-48b2-a24d-c984020d8393"). InnerVolumeSpecName "kube-api-access-pr8l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:42.268443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.268409 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:42.268443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.268446 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c92839-cf49-48b2-a24d-c984020d8393-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:42.268654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.268462 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr8l5\" (UniqueName: \"kubernetes.io/projected/b2c92839-cf49-48b2-a24d-c984020d8393-kube-api-access-pr8l5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:42.881405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.881328 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" Nov 03 20:16:42.881405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.881367 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6" event={"ID":"b2c92839-cf49-48b2-a24d-c984020d8393","Type":"ContainerDied","Data":"3f98581098b6b3907cea33a809a56e27ba91dc4dd7430b947da45c5901cde2b4"} Nov 03 20:16:42.881633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:42.881424 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f98581098b6b3907cea33a809a56e27ba91dc4dd7430b947da45c5901cde2b4" Nov 03 20:16:43.239864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.239831 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn"] Nov 03 20:16:43.240370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.240355 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b2c92839-cf49-48b2-a24d-c984020d8393" containerName="revoke-keys" Nov 03 20:16:43.240413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.240372 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c92839-cf49-48b2-a24d-c984020d8393" containerName="revoke-keys" Nov 03 20:16:43.240494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.240486 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c92839-cf49-48b2-a24d-c984020d8393" containerName="revoke-keys" Nov 03 20:16:43.244093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.244077 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.249176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.249151 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:43.249280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.249180 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:16:43.251288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.251267 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn"] Nov 03 20:16:43.383399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.383370 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.383601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.383425 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.383678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.383595 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rls\" (UniqueName: \"kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.485617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.485584 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.485757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.485668 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.485757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.485744 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rls\" (UniqueName: \"kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.486384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.486360 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.488380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.488354 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.498118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.498052 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rls\" (UniqueName: \"kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls\") pod \"maas-api-revoke-sub-c28d919b-jddqn\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.564986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.564961 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:43.851296 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:43.851219 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:16:43.909138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:43.909108 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn"] Nov 03 20:16:43.909687 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:16:43.909663 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3538e1a_50db_4421_a7bd_460e41b153c9.slice/crio-100d185176f08052037828ab720d8d1b58c9de5c10f2d9059d929d20ec6326a4 WatchSource:0}: Error finding container 100d185176f08052037828ab720d8d1b58c9de5c10f2d9059d929d20ec6326a4: Status 404 returned error can't find the container with id 100d185176f08052037828ab720d8d1b58c9de5c10f2d9059d929d20ec6326a4 Nov 03 20:16:44.521475 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:44.521444 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:44.813825 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:44.813743 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:44.901990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:44.901959 5815 generic.go:356] "Generic (PLEG): container finished" podID="e3538e1a-50db-4421-a7bd-460e41b153c9" containerID="9eb03d8f25241b1298419276944ba30d2e63ddf7987f80447612e7ec43f37f2b" exitCode=0 Nov 03 20:16:44.902133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:44.902048 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" event={"ID":"e3538e1a-50db-4421-a7bd-460e41b153c9","Type":"ContainerDied","Data":"9eb03d8f25241b1298419276944ba30d2e63ddf7987f80447612e7ec43f37f2b"} Nov 03 20:16:44.902133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:44.902095 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" event={"ID":"e3538e1a-50db-4421-a7bd-460e41b153c9","Type":"ContainerStarted","Data":"100d185176f08052037828ab720d8d1b58c9de5c10f2d9059d929d20ec6326a4"} Nov 03 20:16:44.939544 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:44.939497 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:44.940034 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:44.940013 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:46.108895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.108869 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:46.247445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.247414 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls\" (UniqueName: \"kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls\") pod \"e3538e1a-50db-4421-a7bd-460e41b153c9\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " Nov 03 20:16:46.247657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.247590 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca\") pod \"e3538e1a-50db-4421-a7bd-460e41b153c9\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " Nov 03 20:16:46.247657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.247630 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca\") pod \"e3538e1a-50db-4421-a7bd-460e41b153c9\" (UID: \"e3538e1a-50db-4421-a7bd-460e41b153c9\") " Nov 03 20:16:46.248061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.248034 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca" pod "e3538e1a-50db-4421-a7bd-460e41b153c9" (UID: "e3538e1a-50db-4421-a7bd-460e41b153c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:46.249768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.249747 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca" pod "e3538e1a-50db-4421-a7bd-460e41b153c9" (UID: "e3538e1a-50db-4421-a7bd-460e41b153c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:46.250060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.250044 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls" pod "e3538e1a-50db-4421-a7bd-460e41b153c9" (UID: "e3538e1a-50db-4421-a7bd-460e41b153c9"). InnerVolumeSpecName "kube-api-access-p8rls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:46.368595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.367326 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.368595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.367363 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3538e1a-50db-4421-a7bd-460e41b153c9-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.368595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.367378 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rls\" (UniqueName: \"kubernetes.io/projected/e3538e1a-50db-4421-a7bd-460e41b153c9-kube-api-access-p8rls\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.942368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.942295 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" Nov 03 20:16:46.942539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.942289 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn" event={"ID":"e3538e1a-50db-4421-a7bd-460e41b153c9","Type":"ContainerDied","Data":"100d185176f08052037828ab720d8d1b58c9de5c10f2d9059d929d20ec6326a4"} Nov 03 20:16:46.942539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:46.942413 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="100d185176f08052037828ab720d8d1b58c9de5c10f2d9059d929d20ec6326a4" Nov 03 20:16:47.251300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:47.251264 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:16:48.148450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.148415 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8"] Nov 03 20:16:48.149029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.149010 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e3538e1a-50db-4421-a7bd-460e41b153c9" containerName="revoke-keys" Nov 03 20:16:48.149147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.149030 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3538e1a-50db-4421-a7bd-460e41b153c9" containerName="revoke-keys" Nov 03 20:16:48.149212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.149192 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3538e1a-50db-4421-a7bd-460e41b153c9" containerName="revoke-keys" Nov 03 20:16:48.153089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.153067 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.164395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.164363 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8"] Nov 03 20:16:48.190758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.190722 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:48.190892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.190735 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:16:48.201099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.201065 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.201259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.201120 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtjm\" (UniqueName: \"kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.201329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.201274 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.306671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.306250 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.306671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.306388 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.306671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.306425 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtjm\" (UniqueName: \"kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.314526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.312163 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.338215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.338183 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtjm\" (UniqueName: \"kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.370536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.344449 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a56be763-vzbt8\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:48.470799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:48.470140 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:49.084801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:49.069387 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8"] Nov 03 20:16:49.089830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:49.089749 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" event={"ID":"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca","Type":"ContainerStarted","Data":"c570b1837b1fc9bd6de103c81ed512924a7cec57a8473475eb893689e6692c16"} Nov 03 20:16:49.824236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:49.823652 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:49.938638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:49.937633 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.194583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.193773 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" event={"ID":"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca","Type":"ContainerStarted","Data":"b39d19cb2f677a10c1eb2a8bc544f3e1fb209926d1c53ab14e78ec38f85df0c3"} Nov 03 20:16:50.246534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.226366 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" podStartSLOduration=2.2263446 podStartE2EDuration="2.2263446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:50.224299045 +0000 UTC m=+919.754644719" watchObservedRunningTime="2026-11-03 20:16:50.2263446 +0000 UTC m=+919.756690272" Nov 03 20:16:50.352533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.351605 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.436133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.435712 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.793595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.785755 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.940299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.940249 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.941339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:50.941318 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:16:51.204591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:51.203788 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.261532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:51.257175 5815 generic.go:356] "Generic (PLEG): container finished" podID="5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" containerID="b39d19cb2f677a10c1eb2a8bc544f3e1fb209926d1c53ab14e78ec38f85df0c3" exitCode=0 Nov 03 20:16:51.261532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:51.257374 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" event={"ID":"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca","Type":"ContainerDied","Data":"b39d19cb2f677a10c1eb2a8bc544f3e1fb209926d1c53ab14e78ec38f85df0c3"} Nov 03 20:16:51.391534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:51.362913 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.391534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:51.364866 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:16:52.204536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:52.171597 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:52.652322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:52.231300 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:52.652941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:52.364598 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:16:52.786713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:52.759260 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:16:52.953587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:52.898247 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.031637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.031588 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:16:53.288645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.288606 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:53.289218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.289185 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.456523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.456465 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:53.456828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.456797 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.508531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.488616 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:53.508531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.488683 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.661209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.661116 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.673539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.661751 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.735562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.674210 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.964960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.956017 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:53.964960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.956070 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.987013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.976916 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:53.987013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:53.976974 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:54.033179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.033131 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:16:54.034105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.034082 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:16:54.215574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.214867 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:16:54.404749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.404706 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7"] Nov 03 20:16:54.504596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.499392 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-nj4bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:54.504596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.499455 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:54.517529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.513232 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:54.517529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.513298 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:54.751531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.735666 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:54.851532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.836938 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z774r\" (UniqueName: \"kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:54.851532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.837071 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:54.851532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.837121 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:54.881805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.881774 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:54.901455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:54.882347 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:54.901455 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:54.893155 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:55.087083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.086975 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:55.115535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.092559 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca\") pod \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " Nov 03 20:16:55.115889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.115858 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca\") pod \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " Nov 03 20:16:55.116133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.116116 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm\" (UniqueName: \"kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm\") pod \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\" (UID: \"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca\") " Nov 03 20:16:55.125946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.102139 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca" pod "5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" (UID: "5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:55.145090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.145053 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.145219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.145128 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.145352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.145333 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z774r\" (UniqueName: \"kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.145469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.145442 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:55.146491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.146461 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.158407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.158385 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" Nov 03 20:16:55.161885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.161859 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca" pod "5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" (UID: "5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:55.186559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.183672 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:55.196228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.196191 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm" pod "5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" (UID: "5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca"). InnerVolumeSpecName "kube-api-access-ndtjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:55.199536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.198386 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z774r\" (UniqueName: \"kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.220541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.215313 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf931bb-9zkq7\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.240374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.240323 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8" event={"ID":"5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca","Type":"ContainerDied","Data":"c570b1837b1fc9bd6de103c81ed512924a7cec57a8473475eb893689e6692c16"} Nov 03 20:16:55.240571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.240397 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c570b1837b1fc9bd6de103c81ed512924a7cec57a8473475eb893689e6692c16" Nov 03 20:16:55.240571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.240419 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7"] Nov 03 20:16:55.240571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.240551 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:16:55.312529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.310446 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:55.312529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.310541 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtjm\" (UniqueName: \"kubernetes.io/projected/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca-kube-api-access-ndtjm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:55.488843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.488812 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:55.700765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.700735 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:16:55.848401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:55.848298 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:16:56.247115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:56.247078 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7"] Nov 03 20:16:57.222625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:57.222590 5815 generic.go:356] "Generic (PLEG): container finished" podID="54a22b04-cb03-4be6-a415-536e9cecac10" containerID="f319341c5d6e43455fc01146117e8a4d63828132cb3884a8670a0017806f9e46" exitCode=0 Nov 03 20:16:57.223003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:57.222676 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" event={"ID":"54a22b04-cb03-4be6-a415-536e9cecac10","Type":"ContainerDied","Data":"f319341c5d6e43455fc01146117e8a4d63828132cb3884a8670a0017806f9e46"} Nov 03 20:16:57.223003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:57.222735 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" event={"ID":"54a22b04-cb03-4be6-a415-536e9cecac10","Type":"ContainerStarted","Data":"7259d43ad7b29a86c7e8aa4f2b0ccef84a1218029f3d4c03eba73e4b80cee6aa"} Nov 03 20:16:58.375904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.375879 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:58.471070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.471043 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca\") pod \"54a22b04-cb03-4be6-a415-536e9cecac10\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " Nov 03 20:16:58.471243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.471169 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca\") pod \"54a22b04-cb03-4be6-a415-536e9cecac10\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " Nov 03 20:16:58.471243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.471209 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r\" (UniqueName: \"kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r\") pod \"54a22b04-cb03-4be6-a415-536e9cecac10\" (UID: \"54a22b04-cb03-4be6-a415-536e9cecac10\") " Nov 03 20:16:58.471448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.471420 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca" pod "54a22b04-cb03-4be6-a415-536e9cecac10" (UID: "54a22b04-cb03-4be6-a415-536e9cecac10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:58.471671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.471651 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:58.473464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.473442 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca" pod "54a22b04-cb03-4be6-a415-536e9cecac10" (UID: "54a22b04-cb03-4be6-a415-536e9cecac10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:58.473751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.473726 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r" pod "54a22b04-cb03-4be6-a415-536e9cecac10" (UID: "54a22b04-cb03-4be6-a415-536e9cecac10"). InnerVolumeSpecName "kube-api-access-z774r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:58.572940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.572863 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a22b04-cb03-4be6-a415-536e9cecac10-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:58.572940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:58.572894 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z774r\" (UniqueName: \"kubernetes.io/projected/54a22b04-cb03-4be6-a415-536e9cecac10-kube-api-access-z774r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:59.241464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:59.241436 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" Nov 03 20:16:59.241686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:59.241434 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7" event={"ID":"54a22b04-cb03-4be6-a415-536e9cecac10","Type":"ContainerDied","Data":"7259d43ad7b29a86c7e8aa4f2b0ccef84a1218029f3d4c03eba73e4b80cee6aa"} Nov 03 20:16:59.241686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:16:59.241579 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7259d43ad7b29a86c7e8aa4f2b0ccef84a1218029f3d4c03eba73e4b80cee6aa" Nov 03 20:16:59.855109 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:16:59.855072 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:17:00.384065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.384027 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47"] Nov 03 20:17:00.384995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.384969 5815 cpu_manager.go:414] "containerMap: removing container" podUID="54a22b04-cb03-4be6-a415-536e9cecac10" containerName="revoke-keys" Nov 03 20:17:00.384995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.384995 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a22b04-cb03-4be6-a415-536e9cecac10" containerName="revoke-keys" Nov 03 20:17:00.385202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.385035 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" containerName="revoke-keys" Nov 03 20:17:00.385202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.385045 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" containerName="revoke-keys" Nov 03 20:17:00.385322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.385222 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" containerName="revoke-keys" Nov 03 20:17:00.385322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.385247 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a22b04-cb03-4be6-a415-536e9cecac10" containerName="revoke-keys" Nov 03 20:17:00.389875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.389850 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.405761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.405196 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47"] Nov 03 20:17:00.406391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.406362 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"default-dockercfg-5n6wm\"" Nov 03 20:17:00.406584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.406381 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"kube-root-ca.crt\"" Nov 03 20:17:00.406584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.406452 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"distinct-2-w2-085566-kserve-self-signed-certs\"" Nov 03 20:17:00.426577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.416628 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"openshift-service-ca.crt\"" Nov 03 20:17:00.498940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.498908 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bh5\" (UniqueName: \"kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.499109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.499014 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.499109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.499063 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.499251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.499114 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.499251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.499168 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.499361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.499320 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.606598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.606561 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bh5\" (UniqueName: \"kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.606914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.606892 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.607065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.607051 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.607181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.607167 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.607328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.607312 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.607541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.607525 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.608253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.608215 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.612518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.612480 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.612729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.612700 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.618026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.617760 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.636227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.635563 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.639433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.639376 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bh5\" (UniqueName: \"kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5\") pod \"distinct-2-w2-085566-kserve-69577677d-x8x47\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:00.748104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:00.748078 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:01.078648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.078628 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47"] Nov 03 20:17:01.246547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.246264 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk"] Nov 03 20:17:01.256416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.256381 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.280533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.275949 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:01.281355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.281307 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:17:01.289422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.285370 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk"] Nov 03 20:17:01.296773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.296740 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" event={"ID":"12da41d9-bf1f-4cd6-9993-ba7234f4c732","Type":"ContainerStarted","Data":"3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e"} Nov 03 20:17:01.297049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.297025 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" event={"ID":"12da41d9-bf1f-4cd6-9993-ba7234f4c732","Type":"ContainerStarted","Data":"7acf1185f52a362f794a3c04bfe95b7d9983066b01ff4beb2c2b8001622a2b98"} Nov 03 20:17:01.318354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.318008 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.318354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.318064 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.318354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.318149 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hdn\" (UniqueName: \"kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.319976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.319932 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podStartSLOduration=1.31991868 podStartE2EDuration="1.31991868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:01.317375149 +0000 UTC m=+930.847720818" watchObservedRunningTime="2026-11-03 20:17:01.31991868 +0000 UTC m=+930.850264352" Nov 03 20:17:01.418912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.418824 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.419075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.418919 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hdn\" (UniqueName: \"kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.419138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.419079 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.419671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.419640 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.422024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.422000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.473492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.466391 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hdn\" (UniqueName: \"kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn\") pod \"maas-api-revoke-sub-7b221c5e-4kbbk\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.621570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.621527 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:01.748814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:01.748291 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:02.112826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:02.112777 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk"] Nov 03 20:17:02.308556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:02.306052 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" event={"ID":"3674f5f6-5741-4844-b774-365b8943e79c","Type":"ContainerStarted","Data":"47442aa3da6bc0afecc7f40c3af7a55227463934041eedad2e01b7c6517351b1"} Nov 03 20:17:03.318864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:03.318830 5815 generic.go:356] "Generic (PLEG): container finished" podID="3674f5f6-5741-4844-b774-365b8943e79c" containerID="f502b0c3f9de7cc54ec3a5552fc57962235409bd8a7586027645a4dab7fb7c8e" exitCode=0 Nov 03 20:17:03.319323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:03.318870 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" event={"ID":"3674f5f6-5741-4844-b774-365b8943e79c","Type":"ContainerDied","Data":"f502b0c3f9de7cc54ec3a5552fc57962235409bd8a7586027645a4dab7fb7c8e"} Nov 03 20:17:04.504679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.504645 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:04.585269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.585176 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn\" (UniqueName: \"kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn\") pod \"3674f5f6-5741-4844-b774-365b8943e79c\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " Nov 03 20:17:04.648405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.648347 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn" pod "3674f5f6-5741-4844-b774-365b8943e79c" (UID: "3674f5f6-5741-4844-b774-365b8943e79c"). InnerVolumeSpecName "kube-api-access-q6hdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:04.688566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.688536 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca\") pod \"3674f5f6-5741-4844-b774-365b8943e79c\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " Nov 03 20:17:04.688714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.688592 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca\") pod \"3674f5f6-5741-4844-b774-365b8943e79c\" (UID: \"3674f5f6-5741-4844-b774-365b8943e79c\") " Nov 03 20:17:04.689712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.689299 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hdn\" (UniqueName: \"kubernetes.io/projected/3674f5f6-5741-4844-b774-365b8943e79c-kube-api-access-q6hdn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:04.690089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.690067 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca" pod "3674f5f6-5741-4844-b774-365b8943e79c" (UID: "3674f5f6-5741-4844-b774-365b8943e79c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:04.692489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.692451 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca" pod "3674f5f6-5741-4844-b774-365b8943e79c" (UID: "3674f5f6-5741-4844-b774-365b8943e79c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:04.791683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.791652 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:04.791683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:04.791686 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3674f5f6-5741-4844-b774-365b8943e79c-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:05.206679 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:05.206645 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:17:05.353186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:05.353158 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" Nov 03 20:17:05.353186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:05.353164 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk" event={"ID":"3674f5f6-5741-4844-b774-365b8943e79c","Type":"ContainerDied","Data":"47442aa3da6bc0afecc7f40c3af7a55227463934041eedad2e01b7c6517351b1"} Nov 03 20:17:05.353393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:05.353218 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47442aa3da6bc0afecc7f40c3af7a55227463934041eedad2e01b7c6517351b1" Nov 03 20:17:09.092894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.092853 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4"] Nov 03 20:17:09.093572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.093366 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3674f5f6-5741-4844-b774-365b8943e79c" containerName="revoke-keys" Nov 03 20:17:09.093572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.093376 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3674f5f6-5741-4844-b774-365b8943e79c" containerName="revoke-keys" Nov 03 20:17:09.093572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.093470 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3674f5f6-5741-4844-b774-365b8943e79c" containerName="revoke-keys" Nov 03 20:17:09.097400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.097379 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.104474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.104194 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:09.104474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.104255 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:17:09.109335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.109309 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4"] Nov 03 20:17:09.156302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.156277 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndpq\" (UniqueName: \"kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.156431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.156314 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.156431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.156398 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.257361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.257335 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndpq\" (UniqueName: \"kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.257552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.257373 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.257552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.257419 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.258112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.258092 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.260024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.260007 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.270253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.270227 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndpq\" (UniqueName: \"kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq\") pod \"maas-api-revoke-sub-289c1d61-2x2x4\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.414529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.414436 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:09.849164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:09.849106 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4"] Nov 03 20:17:10.476531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:10.474970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" event={"ID":"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89","Type":"ContainerStarted","Data":"60ba2bbb1f185310a6e0dc572403092bd1b07b087df84987c719315ae47868b7"} Nov 03 20:17:10.476531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:10.475054 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" event={"ID":"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89","Type":"ContainerStarted","Data":"67f81ec01909cba11f435b7b9512bdb28c7113120cbc9e40cd10edc12d48fed3"} Nov 03 20:17:10.527532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:10.509616 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" podStartSLOduration=1.5095961610000002 podStartE2EDuration="1.509596161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:10.497958258 +0000 UTC m=+940.028303933" watchObservedRunningTime="2026-11-03 20:17:10.509596161 +0000 UTC m=+940.039941835" Nov 03 20:17:10.760612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:10.753997 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:11.520609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:11.520542 5815 generic.go:356] "Generic (PLEG): container finished" podID="64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" containerID="60ba2bbb1f185310a6e0dc572403092bd1b07b087df84987c719315ae47868b7" exitCode=0 Nov 03 20:17:11.521860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:11.521407 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" event={"ID":"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89","Type":"ContainerDied","Data":"60ba2bbb1f185310a6e0dc572403092bd1b07b087df84987c719315ae47868b7"} Nov 03 20:17:11.538050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:11.536657 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:11.796591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:11.793615 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:11.872040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:11.866596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.268533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:12.261405 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:12.907588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:12.787687 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.907588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:12.846265 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:13.116534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:13.107677 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:17:13.591008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:13.590964 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:13.758555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:13.758454 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:17:13.850701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:13.850596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:17:13.908533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:13.861615 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:17:14.289547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.283368 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:17:14.519615 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:14.513219 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:17:14.594954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.593495 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:17:14.809538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.806198 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:14.830552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.809764 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:14.901214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.900108 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:17:14.925010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.924978 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:14.949817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.949797 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:17:14.950062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.950044 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv"] Nov 03 20:17:14.974577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:14.964744 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.143312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.141276 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:15.143641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.143585 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.263591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.239849 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca\") pod \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " Nov 03 20:17:15.263591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.239902 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq\" (UniqueName: \"kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq\") pod \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " Nov 03 20:17:15.263591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.239984 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca\") pod \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\" (UID: \"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89\") " Nov 03 20:17:15.264194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.264166 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv"] Nov 03 20:17:15.314544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.264685 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca" pod "64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" (UID: "64dfbd0a-9a5a-4cd0-90ed-d88be7769e89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:15.314544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.269573 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:17:15.314544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.296372 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.349532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.344427 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:15.349532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.345023 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-085566-openshift-default-dockercfg-dt25g\"" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.460677 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.460740 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.460794 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.460874 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.460900 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.460925 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvvt\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.461024 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.461089 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.477808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.461187 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.565671 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.565768 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.565820 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.565866 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.565946 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.565973 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.566000 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvvt\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.566088 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.568534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.566140 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:15.747113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.746733 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq" pod "64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" (UID: "64dfbd0a-9a5a-4cd0-90ed-d88be7769e89"). InnerVolumeSpecName "kube-api-access-vndpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:15.747689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.747654 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca" pod "64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" (UID: "64dfbd0a-9a5a-4cd0-90ed-d88be7769e89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:15.929534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.926012 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:17:15.993564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.965896 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.995708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.995661 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:17:16.029642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:15.965961 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:17:16.029642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.021104 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:17:16.134534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.130907 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:17:16.277767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.277726 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.399670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.299637 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:16.401049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.315601 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.401223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.317521 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.401339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.317786 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.401436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.318010 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.401571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.377125 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:16.401712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.401694 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndpq\" (UniqueName: \"kubernetes.io/projected/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89-kube-api-access-vndpq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:16.494532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.473967 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.730534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.710803 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.899543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.897904 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:16.899543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.898257 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" Nov 03 20:17:16.938381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:16.938348 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvvt\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-j86hv\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:17.038440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.038325 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:17:17.042782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.042743 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:17.074910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.074877 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:17.090866 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:17.090832 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:17:17.131316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.131279 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:17.203099 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:17.195632 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:17:17.203099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.198946 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4" event={"ID":"64dfbd0a-9a5a-4cd0-90ed-d88be7769e89","Type":"ContainerDied","Data":"67f81ec01909cba11f435b7b9512bdb28c7113120cbc9e40cd10edc12d48fed3"} Nov 03 20:17:17.203099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.199000 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f81ec01909cba11f435b7b9512bdb28c7113120cbc9e40cd10edc12d48fed3" Nov 03 20:17:17.203099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.199061 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:17:17.446538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.438904 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:17:17.462962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.462846 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:17:17.464430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.464396 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:17.475580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.464770 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:17:17.518650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.518622 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:17:17.567262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:17.567177 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:17:18.076787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:18.075625 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:18.076787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:18.075722 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:18.076787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:18.075773 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:18.163266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:18.162776 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv"] Nov 03 20:17:19.058224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:19.058181 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" event={"ID":"67b7dce9-2229-44e9-b85f-279a9092dece","Type":"ContainerStarted","Data":"3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251"} Nov 03 20:17:19.058399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:19.058245 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" event={"ID":"67b7dce9-2229-44e9-b85f-279a9092dece","Type":"ContainerStarted","Data":"25c6afe17eaf55711b0f7ab66493e5ebbce57f86cf142c1b35bc67f16975fa99"} Nov 03 20:17:19.092531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:19.090592 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:19.095294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:19.095260 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:19.095560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:19.095533 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:19.100909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:19.096815 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podStartSLOduration=5.096798273 podStartE2EDuration="5.096798273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:19.082154675 +0000 UTC m=+948.612500348" watchObservedRunningTime="2026-11-03 20:17:19.096798273 +0000 UTC m=+948.627143946" Nov 03 20:17:20.088533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:20.080834 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:20.088533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:20.080890 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:21.085894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:21.085857 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:21.086171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:21.086142 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:22.089531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:22.085883 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:22.089531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:22.085944 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:23.088531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:23.085856 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:23.088531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:23.085914 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:24.079531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:24.078877 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:24.079531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:24.078941 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:25.079114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:25.079071 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:17:25.079288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:25.079138 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 20:17:26.087816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:26.087782 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:26.088357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:26.088152 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:26.089293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:26.089271 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:17:27.339389 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:27.339357 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:17:29.798483 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:29.798422 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7bbbae_6a84_49a8_a59b_771e82999294.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:17:31.161481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:31.161437 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:17:34.701532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.696774 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x"] Nov 03 20:17:34.701532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.697619 5815 cpu_manager.go:414] "containerMap: removing container" podUID="64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" containerName="revoke-keys" Nov 03 20:17:34.701532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.697640 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" containerName="revoke-keys" Nov 03 20:17:34.701532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.697807 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" containerName="revoke-keys" Nov 03 20:17:34.717936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.717911 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.821533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.797412 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x"] Nov 03 20:17:34.864535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.857891 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.864535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.858014 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.864535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.858086 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.864535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.858121 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.864535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.858151 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.864535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.858274 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988301 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988414 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988477 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988556 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988641 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988670 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988697 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:34.988854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:34.988720 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszfv\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.019532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.014127 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.019532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.014425 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.141532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.102450 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.142023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.141993 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.142261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.142242 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszfv\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.142454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.128110 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.142600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.103308 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.143067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.118821 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.193530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.144808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.193530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.182899 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.205528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.195931 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszfv\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.205528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.203083 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:35.359591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:35.359432 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:36.160919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.160305 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:36.444580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.439947 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x"] Nov 03 20:17:36.502534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.448736 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:17:36.502534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.472584 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:36.589803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.568465 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:36.589803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.568587 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:36.589803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.568635 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:36.589803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.585874 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:36.841704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:36.841597 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" event={"ID":"1bcc9bc8-8d66-4ef4-bf80-20619126fced","Type":"ContainerStarted","Data":"982d012b2c11c008ecd8876675b8eba984ee709b805cedda844aee34a90b36b4"} Nov 03 20:17:38.275538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:38.275193 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" event={"ID":"1bcc9bc8-8d66-4ef4-bf80-20619126fced","Type":"ContainerStarted","Data":"f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d"} Nov 03 20:17:38.332526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:38.332437 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podStartSLOduration=4.332417378 podStartE2EDuration="4.332417378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:38.326406433 +0000 UTC m=+967.856752103" watchObservedRunningTime="2026-11-03 20:17:38.332417378 +0000 UTC m=+967.862763051" Nov 03 20:17:38.363217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:38.363124 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:38.363593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:38.363553 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:38.363823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:38.363780 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:39.366533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:39.365948 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:39.366533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:39.366071 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:40.361029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:40.360991 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:40.361450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:40.361420 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:41.361281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.361251 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:41.361813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.361320 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:41.910600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.910571 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz"] Nov 03 20:17:41.914856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.914836 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:41.919716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.919697 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:41.919884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.919773 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:17:41.924207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:41.924172 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz"] Nov 03 20:17:42.037549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.031627 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.037549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.031703 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.037549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.031784 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdg59\" (UniqueName: \"kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.142930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.142886 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdg59\" (UniqueName: \"kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.143336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.143314 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.143410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.143379 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.144540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.144476 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.146729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.146705 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.155435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.155414 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdg59\" (UniqueName: \"kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59\") pod \"maas-api-revoke-sub-a6d296a9-dhckz\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.236206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.236182 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:42.360546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.360138 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:42.360546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.360184 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:42.552461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:42.552428 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz"] Nov 03 20:17:42.552851 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:17:42.552495 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183fa9c4_eeb9_46d3_bea0_be98263488d3.slice/crio-58d716d78155a21fa4ee2184120447c91dc37d1721ad6d64e84bd2ca2833f6d5 WatchSource:0}: Error finding container 58d716d78155a21fa4ee2184120447c91dc37d1721ad6d64e84bd2ca2833f6d5: Status 404 returned error can't find the container with id 58d716d78155a21fa4ee2184120447c91dc37d1721ad6d64e84bd2ca2833f6d5 Nov 03 20:17:43.335699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.335653 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" event={"ID":"183fa9c4-eeb9-46d3-bea0-be98263488d3","Type":"ContainerStarted","Data":"ae9357d3e2c222b26e5236d31310a47eb105fd79f598194374e26ddc9f1d3a7b"} Nov 03 20:17:43.335699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.335709 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" event={"ID":"183fa9c4-eeb9-46d3-bea0-be98263488d3","Type":"ContainerStarted","Data":"58d716d78155a21fa4ee2184120447c91dc37d1721ad6d64e84bd2ca2833f6d5"} Nov 03 20:17:43.358369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.358308 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" podStartSLOduration=2.358289053 podStartE2EDuration="2.358289053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:43.354579123 +0000 UTC m=+972.884924796" watchObservedRunningTime="2026-11-03 20:17:43.358289053 +0000 UTC m=+972.888634726" Nov 03 20:17:43.362297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.362244 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:43.362459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.362316 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:43.431374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.429368 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf"] Nov 03 20:17:43.456013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.445342 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.456538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.456392 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf"] Nov 03 20:17:43.600592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.600262 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zfmr\" (UniqueName: \"kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.600592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.600479 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.600592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.600585 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.729537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.729159 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zfmr\" (UniqueName: \"kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.729537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.729446 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.729815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.729567 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.730541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.730477 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.735767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.735716 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.743836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.743806 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zfmr\" (UniqueName: \"kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr\") pod \"maas-api-revoke-sub-b679c5a5-tftwf\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.818095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:43.818072 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:43.856196 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:17:43.856111 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:17:44.130753 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:17:44.130719 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6f982a_4ea4_4b7e_a4c2_77b261796484.slice/crio-a5e758bb90beccbfc46c50140ac2f26fffa6cf46faff6bc36b987e9e27199743 WatchSource:0}: Error finding container a5e758bb90beccbfc46c50140ac2f26fffa6cf46faff6bc36b987e9e27199743: Status 404 returned error can't find the container with id a5e758bb90beccbfc46c50140ac2f26fffa6cf46faff6bc36b987e9e27199743 Nov 03 20:17:44.133919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.133894 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf"] Nov 03 20:17:44.351487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.351455 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" event={"ID":"ac6f982a-4ea4-4b7e-a4c2-77b261796484","Type":"ContainerStarted","Data":"a5e758bb90beccbfc46c50140ac2f26fffa6cf46faff6bc36b987e9e27199743"} Nov 03 20:17:44.353744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.353717 5815 generic.go:356] "Generic (PLEG): container finished" podID="183fa9c4-eeb9-46d3-bea0-be98263488d3" containerID="ae9357d3e2c222b26e5236d31310a47eb105fd79f598194374e26ddc9f1d3a7b" exitCode=0 Nov 03 20:17:44.354139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.354117 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" event={"ID":"183fa9c4-eeb9-46d3-bea0-be98263488d3","Type":"ContainerDied","Data":"ae9357d3e2c222b26e5236d31310a47eb105fd79f598194374e26ddc9f1d3a7b"} Nov 03 20:17:44.360926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.360904 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:17:44.361126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.361103 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 20:17:44.418923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.418314 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67"] Nov 03 20:17:44.425302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.425279 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.434691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.434662 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67"] Nov 03 20:17:44.566476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566441 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.566660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566489 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthmx\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.566660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566634 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.566803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566680 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.566803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566707 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.566922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566862 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.566922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.566913 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.567103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.567074 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.567242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.567133 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.668723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.668616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.668826 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthmx\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.668886 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.668919 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.668947 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.669001 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.669057 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.669104 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.669157 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.670135 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.670181 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.670457 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.670709 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.670857 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.673395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.673032 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.674872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.674837 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.683026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.683007 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthmx\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.683256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.682964 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-9kx67\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:44.750643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:44.750609 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:45.093708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.093680 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67"] Nov 03 20:17:45.094195 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:17:45.094170 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac25f1d_3097_426f_a9be_c5c1baaacb3e.slice/crio-f87450aecaf717c2940bb78b1270292e8f808a1f067b1ac03579a187d872e736 WatchSource:0}: Error finding container f87450aecaf717c2940bb78b1270292e8f808a1f067b1ac03579a187d872e736: Status 404 returned error can't find the container with id f87450aecaf717c2940bb78b1270292e8f808a1f067b1ac03579a187d872e736 Nov 03 20:17:45.097465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.097422 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:45.097587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.097534 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:45.097587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.097580 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:45.365017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.364927 5815 generic.go:356] "Generic (PLEG): container finished" podID="ac6f982a-4ea4-4b7e-a4c2-77b261796484" containerID="67a8a799efe3b54c30281f590c19663639f6539ce6f60036c6df57a5753c78ab" exitCode=0 Nov 03 20:17:45.365190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.365057 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" event={"ID":"ac6f982a-4ea4-4b7e-a4c2-77b261796484","Type":"ContainerDied","Data":"67a8a799efe3b54c30281f590c19663639f6539ce6f60036c6df57a5753c78ab"} Nov 03 20:17:45.368323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.368248 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" event={"ID":"0ac25f1d-3097-426f-a9be-c5c1baaacb3e","Type":"ContainerStarted","Data":"c7e0a81726d944564212c8ee1d5d83177f31a8d76e8eeefbd934c4c6a00bb202"} Nov 03 20:17:45.368323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.368293 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" event={"ID":"0ac25f1d-3097-426f-a9be-c5c1baaacb3e","Type":"ContainerStarted","Data":"f87450aecaf717c2940bb78b1270292e8f808a1f067b1ac03579a187d872e736"} Nov 03 20:17:45.438267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.437901 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podStartSLOduration=1.437882103 podStartE2EDuration="1.437882103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:45.435690692 +0000 UTC m=+974.966036390" watchObservedRunningTime="2026-11-03 20:17:45.437882103 +0000 UTC m=+974.968227776" Nov 03 20:17:45.639652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.639581 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:45.753139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.753111 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:45.824016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.822348 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59\" (UniqueName: \"kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59\") pod \"183fa9c4-eeb9-46d3-bea0-be98263488d3\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " Nov 03 20:17:45.824016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.822447 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca\") pod \"183fa9c4-eeb9-46d3-bea0-be98263488d3\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " Nov 03 20:17:45.824016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.822715 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca\") pod \"183fa9c4-eeb9-46d3-bea0-be98263488d3\" (UID: \"183fa9c4-eeb9-46d3-bea0-be98263488d3\") " Nov 03 20:17:45.824638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.824610 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca" pod "183fa9c4-eeb9-46d3-bea0-be98263488d3" (UID: "183fa9c4-eeb9-46d3-bea0-be98263488d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:45.845162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.845128 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59" pod "183fa9c4-eeb9-46d3-bea0-be98263488d3" (UID: "183fa9c4-eeb9-46d3-bea0-be98263488d3"). InnerVolumeSpecName "kube-api-access-hdg59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:45.851136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.851082 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca" pod "183fa9c4-eeb9-46d3-bea0-be98263488d3" (UID: "183fa9c4-eeb9-46d3-bea0-be98263488d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:45.859205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.859175 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:45.860969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.860948 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:45.862300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.862283 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:17:45.924259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.924183 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:45.924436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.924423 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdg59\" (UniqueName: \"kubernetes.io/projected/183fa9c4-eeb9-46d3-bea0-be98263488d3-kube-api-access-hdg59\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:45.924537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:45.924500 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/183fa9c4-eeb9-46d3-bea0-be98263488d3-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:46.402562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.402335 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" Nov 03 20:17:46.402562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.402343 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz" event={"ID":"183fa9c4-eeb9-46d3-bea0-be98263488d3","Type":"ContainerDied","Data":"58d716d78155a21fa4ee2184120447c91dc37d1721ad6d64e84bd2ca2833f6d5"} Nov 03 20:17:46.402562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.402402 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d716d78155a21fa4ee2184120447c91dc37d1721ad6d64e84bd2ca2833f6d5" Nov 03 20:17:46.654582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.646418 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:46.736247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.736220 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr\" (UniqueName: \"kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr\") pod \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " Nov 03 20:17:46.736473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.736454 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca\") pod \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " Nov 03 20:17:46.736704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.736685 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca\") pod \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\" (UID: \"ac6f982a-4ea4-4b7e-a4c2-77b261796484\") " Nov 03 20:17:46.737026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.737004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca" pod "ac6f982a-4ea4-4b7e-a4c2-77b261796484" (UID: "ac6f982a-4ea4-4b7e-a4c2-77b261796484"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:46.737483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.737464 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:46.739641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.739617 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca" pod "ac6f982a-4ea4-4b7e-a4c2-77b261796484" (UID: "ac6f982a-4ea4-4b7e-a4c2-77b261796484"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:46.740613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.740594 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr" pod "ac6f982a-4ea4-4b7e-a4c2-77b261796484" (UID: "ac6f982a-4ea4-4b7e-a4c2-77b261796484"). InnerVolumeSpecName "kube-api-access-4zfmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:46.753776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.753734 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:46.754102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.753786 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:46.838466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.838431 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zfmr\" (UniqueName: \"kubernetes.io/projected/ac6f982a-4ea4-4b7e-a4c2-77b261796484-kube-api-access-4zfmr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:46.838466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:46.838460 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6f982a-4ea4-4b7e-a4c2-77b261796484-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:47.424792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:47.424764 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" Nov 03 20:17:47.425029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:47.424823 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf" event={"ID":"ac6f982a-4ea4-4b7e-a4c2-77b261796484","Type":"ContainerDied","Data":"a5e758bb90beccbfc46c50140ac2f26fffa6cf46faff6bc36b987e9e27199743"} Nov 03 20:17:47.425162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:47.425149 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e758bb90beccbfc46c50140ac2f26fffa6cf46faff6bc36b987e9e27199743" Nov 03 20:17:47.753127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:47.753084 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:47.753295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:47.753157 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:48.753693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:48.753656 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:48.754176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:48.753730 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:49.708610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.708579 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v"] Nov 03 20:17:49.709128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.709114 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ac6f982a-4ea4-4b7e-a4c2-77b261796484" containerName="revoke-keys" Nov 03 20:17:49.709175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.709129 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6f982a-4ea4-4b7e-a4c2-77b261796484" containerName="revoke-keys" Nov 03 20:17:49.709175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.709155 5815 cpu_manager.go:414] "containerMap: removing container" podUID="183fa9c4-eeb9-46d3-bea0-be98263488d3" containerName="revoke-keys" Nov 03 20:17:49.709175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.709161 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="183fa9c4-eeb9-46d3-bea0-be98263488d3" containerName="revoke-keys" Nov 03 20:17:49.709272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.709251 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6f982a-4ea4-4b7e-a4c2-77b261796484" containerName="revoke-keys" Nov 03 20:17:49.709272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.709262 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="183fa9c4-eeb9-46d3-bea0-be98263488d3" containerName="revoke-keys" Nov 03 20:17:49.713281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.713257 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.728176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.728149 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v"] Nov 03 20:17:49.753103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.753080 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:49.753224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.753129 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:49.849449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849455 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849521 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849601 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849700 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849716 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849787 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849854 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5fm\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.849896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.849885 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951003 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951040 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951150 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951349 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951377 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951465 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951539 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5fm\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951571 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951712 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.951855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.951796 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.952200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.952173 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.952370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.952344 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.952538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.952490 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.953724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.953694 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.954319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.954290 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.964274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.964251 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:49.980530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:49.973563 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5fm\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:50.032218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.032196 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:50.378682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.378596 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v"] Nov 03 20:17:50.386139 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:17:50.386086 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaf8a64_6829_4034_89f8_a2608c2b7cc0.slice/crio-4fbd328281a0dadadc2bd0fa1d53eb6ce6cb986b3124a197d29b5a5975c21967 WatchSource:0}: Error finding container 4fbd328281a0dadadc2bd0fa1d53eb6ce6cb986b3124a197d29b5a5975c21967: Status 404 returned error can't find the container with id 4fbd328281a0dadadc2bd0fa1d53eb6ce6cb986b3124a197d29b5a5975c21967 Nov 03 20:17:50.427597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.405065 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:50.427597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.405150 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:50.427597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.405190 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:17:50.455217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.455185 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" event={"ID":"1eaf8a64-6829-4034-89f8-a2608c2b7cc0","Type":"ContainerStarted","Data":"4fbd328281a0dadadc2bd0fa1d53eb6ce6cb986b3124a197d29b5a5975c21967"} Nov 03 20:17:50.754373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.754337 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:50.754707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:50.754666 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:51.546050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:51.546010 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" event={"ID":"1eaf8a64-6829-4034-89f8-a2608c2b7cc0","Type":"ContainerStarted","Data":"5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a"} Nov 03 20:17:51.590540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:51.589086 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podStartSLOduration=2.589066297 podStartE2EDuration="2.589066297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:51.58303996 +0000 UTC m=+981.113385633" watchObservedRunningTime="2026-11-03 20:17:51.589066297 +0000 UTC m=+981.119411971" Nov 03 20:17:51.752992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:51.752957 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:51.753198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:51.753017 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:52.033145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:52.033116 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:52.035106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:52.035074 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:17:52.035228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:52.035125 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:17:52.753429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:52.753396 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:17:52.753945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:52.753455 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:17:53.033580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.033472 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:17:53.033580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.033558 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:17:53.652326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.652292 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl"] Nov 03 20:17:53.656666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.656645 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.662012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.661978 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:53.662130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.662025 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:17:53.697064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.673240 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl"] Nov 03 20:17:53.720336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.719572 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpf6m\" (UniqueName: \"kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.720336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.719715 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.720336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.719820 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.757962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.757933 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:53.758314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.758271 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:53.758972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.758952 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:17:53.821815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.821778 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpf6m\" (UniqueName: \"kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.822951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.822060 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.822951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.822161 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.822951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.822905 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.833902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.833874 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:53.846397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:53.846376 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpf6m\" (UniqueName: \"kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m\") pod \"maas-api-revoke-sub-bcc55453-fxkpl\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:54.026967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:54.026938 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:54.033582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:54.033556 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:17:54.033685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:54.033602 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:17:54.348296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:54.346186 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl"] Nov 03 20:17:54.348296 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:17:54.346782 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f96db3a_5e40_413a_8fd3_ac5eba79b2c1.slice/crio-9bc555d0896759cf1c8a2f49b1c06adb6b752ac53fdb8c9ebccbf17d83d95b5b WatchSource:0}: Error finding container 9bc555d0896759cf1c8a2f49b1c06adb6b752ac53fdb8c9ebccbf17d83d95b5b: Status 404 returned error can't find the container with id 9bc555d0896759cf1c8a2f49b1c06adb6b752ac53fdb8c9ebccbf17d83d95b5b Nov 03 20:17:54.721716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:54.720700 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" event={"ID":"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1","Type":"ContainerStarted","Data":"9bc555d0896759cf1c8a2f49b1c06adb6b752ac53fdb8c9ebccbf17d83d95b5b"} Nov 03 20:17:55.033293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:55.033263 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:17:55.033753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:55.033318 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:17:55.784607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:55.784499 5815 generic.go:356] "Generic (PLEG): container finished" podID="9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" containerID="f456d1cbd6b6ef4864c83f02148325ef19023fbe1533e886379cc1aa21722b60" exitCode=0 Nov 03 20:17:55.786189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:55.786159 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" event={"ID":"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1","Type":"ContainerDied","Data":"f456d1cbd6b6ef4864c83f02148325ef19023fbe1533e886379cc1aa21722b60"} Nov 03 20:17:56.033129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:56.033093 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:17:56.033327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:56.033155 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:17:57.032843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.032816 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:17:57.032966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.032874 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:17:57.036088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.036068 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:57.105472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.105366 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m\" (UniqueName: \"kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m\") pod \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " Nov 03 20:17:57.105731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.105702 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca\") pod \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " Nov 03 20:17:57.106591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.105823 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca\") pod \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\" (UID: \"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1\") " Nov 03 20:17:57.106962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.106900 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca" pod "9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" (UID: "9f96db3a-5e40-413a-8fd3-ac5eba79b2c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:57.114718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.114670 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca" pod "9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" (UID: "9f96db3a-5e40-413a-8fd3-ac5eba79b2c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:57.114718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.114677 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m" pod "9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" (UID: "9f96db3a-5e40-413a-8fd3-ac5eba79b2c1"). InnerVolumeSpecName "kube-api-access-hpf6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:57.207352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.207317 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:57.207352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.207351 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:57.207352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.207362 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpf6m\" (UniqueName: \"kubernetes.io/projected/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1-kube-api-access-hpf6m\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:57.819445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.819415 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" Nov 03 20:17:57.819445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.819438 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl" event={"ID":"9f96db3a-5e40-413a-8fd3-ac5eba79b2c1","Type":"ContainerDied","Data":"9bc555d0896759cf1c8a2f49b1c06adb6b752ac53fdb8c9ebccbf17d83d95b5b"} Nov 03 20:17:57.819666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:57.819487 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc555d0896759cf1c8a2f49b1c06adb6b752ac53fdb8c9ebccbf17d83d95b5b" Nov 03 20:17:58.626045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:58.626016 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:58.627236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:58.627211 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:17:58.628157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:17:58.628129 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 20:18:01.787780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.787747 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5"] Nov 03 20:18:01.788345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.788328 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" containerName="revoke-keys" Nov 03 20:18:01.788395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.788347 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" containerName="revoke-keys" Nov 03 20:18:01.788465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.788455 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" containerName="revoke-keys" Nov 03 20:18:01.792330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.792313 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.806260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.806229 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\"" Nov 03 20:18:01.809383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.809356 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5"] Nov 03 20:18:01.934266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934234 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934265 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934286 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxsw\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934428 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934537 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934586 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934644 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934675 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:01.934866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:01.934764 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036031 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036063 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036082 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxsw\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036128 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036169 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036203 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036283 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036326 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036365 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036490 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036571 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036608 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.036859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.036836 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.037162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.037113 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.038935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.038874 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.039040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.038968 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.051150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.051121 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.051494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.051473 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxsw\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.109036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.109004 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:02.483521 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:18:02.483476 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16ffb8c_97a0_4696_8a83_80d14a4ebab8.slice/crio-60a577176a65376d5b5114b0efcf1bce63f225b99fc521946c71a74063ba9a76 WatchSource:0}: Error finding container 60a577176a65376d5b5114b0efcf1bce63f225b99fc521946c71a74063ba9a76: Status 404 returned error can't find the container with id 60a577176a65376d5b5114b0efcf1bce63f225b99fc521946c71a74063ba9a76 Nov 03 20:18:02.483746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.483726 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5"] Nov 03 20:18:02.486065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.486029 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:02.486162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.486096 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:02.486162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.486127 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:02.890764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.890674 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" event={"ID":"e16ffb8c-97a0-4696-8a83-80d14a4ebab8","Type":"ContainerStarted","Data":"5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887"} Nov 03 20:18:02.891381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.891352 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" event={"ID":"e16ffb8c-97a0-4696-8a83-80d14a4ebab8","Type":"ContainerStarted","Data":"60a577176a65376d5b5114b0efcf1bce63f225b99fc521946c71a74063ba9a76"} Nov 03 20:18:02.921099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:02.921010 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podStartSLOduration=1.920987773 podStartE2EDuration="1.920987773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:02.915164488 +0000 UTC m=+992.445510285" watchObservedRunningTime="2026-11-03 20:18:02.920987773 +0000 UTC m=+992.451333517" Nov 03 20:18:03.109741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:03.109698 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:03.111676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:03.111645 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:18:03.111805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:03.111694 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 20:18:04.110340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:04.110303 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:18:04.110855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:04.110398 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 20:18:05.114241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:05.114162 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:18:05.114996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:05.114961 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 20:18:06.110090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.110042 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:18:06.110269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.110103 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 20:18:06.220244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.220212 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564"] Nov 03 20:18:06.226402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.226380 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.246607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.246586 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-98148a-openshift-default-dockercfg-hhl82\"" Nov 03 20:18:06.249420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.249279 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564"] Nov 03 20:18:06.311559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.311471 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.311559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.311562 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.311804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.311643 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.311863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.311805 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.311863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.311852 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.311967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.311910 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.312211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.312186 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.312319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.312241 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9kxw\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.312375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.312336 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417526 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417591 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417653 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417761 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417793 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9kxw\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417875 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.417961 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.418002 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.418069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.418733 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.425665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.420485 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.445098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.444677 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.445919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.445589 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.445919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.445867 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.465114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.464271 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.466940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.466890 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.480570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.480545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9kxw\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.496237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.496168 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo\") pod \"e2e-api-a-98148a-openshift-default-fbb48db79-cd564\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:06.551118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:06.550779 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:07.067530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.067277 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564"] Nov 03 20:18:07.073714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.073672 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:07.074868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.074841 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:07.079634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.079604 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:07.110100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.110074 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:18:07.110214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.110154 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 20:18:07.978693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:07.978653 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" event={"ID":"46128744-78a6-4cef-b813-11b4202b3aab","Type":"ContainerStarted","Data":"00e356df1ee7e5a33010c4d7cc375e580a87d437468ed3f3bf75c653e7b8e353"} Nov 03 20:18:08.010108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.010070 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" event={"ID":"46128744-78a6-4cef-b813-11b4202b3aab","Type":"ContainerStarted","Data":"70e5915b9c1f2c2fe6172e48d2f621947fe006906cc4ecfa922b458a979baf00"} Nov 03 20:18:08.061056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.057102 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podStartSLOduration=2.057080018 podStartE2EDuration="2.057080018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:08.040754206 +0000 UTC m=+997.571099890" watchObservedRunningTime="2026-11-03 20:18:08.057080018 +0000 UTC m=+997.587425692" Nov 03 20:18:08.111388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.109839 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:18:08.111388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.109887 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 03 20:18:08.551604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.551526 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:08.558767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.558745 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:08.989370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.989329 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:08.990248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:08.990227 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:18:09.113766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:09.113738 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:09.114163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:09.114135 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:09.114783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:09.114766 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:13.852983 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:18:13.852946 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:18:18.571581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:18.571545 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:18:20.919747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.919708 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c"] Nov 03 20:18:20.929454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.928120 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:20.956697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.943579 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9b65143e\"/\"openshift-service-ca.crt\"" Nov 03 20:18:20.956697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.943828 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9b65143e\"/\"e2e-lifecycle-model-9b65143e-kserve-self-signed-certs\"" Nov 03 20:18:20.956697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.943897 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9b65143e\"/\"default-dockercfg-c7tn5\"" Nov 03 20:18:20.956697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.944041 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9b65143e\"/\"kube-root-ca.crt\"" Nov 03 20:18:20.962424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.962015 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:20.962424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.962068 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:20.962424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.962174 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:20.962424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.962254 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:20.962424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.962284 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:20.962424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:20.962342 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psgct\" (UniqueName: \"kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.493425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.492812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.493425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.492888 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.493425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.493029 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.493425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.493135 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.493425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.493172 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.493425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.493239 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psgct\" (UniqueName: \"kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.555254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.486419 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c"] Nov 03 20:18:21.556380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.556355 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.605532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.593297 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.691545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.626865 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.691545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.647144 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.692186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.692116 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.698228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.698143 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psgct\" (UniqueName: \"kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct\") pod \"e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:21.715651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:21.715629 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:22.486532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:22.484543 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:22.809152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:22.808728 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c"] Nov 03 20:18:23.838786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:23.837448 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" event={"ID":"6ea24e8f-5b74-4ddd-b7fc-810009660e94","Type":"ContainerStarted","Data":"b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c"} Nov 03 20:18:23.838786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:23.837498 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" event={"ID":"6ea24e8f-5b74-4ddd-b7fc-810009660e94","Type":"ContainerStarted","Data":"5c82d8b2fc600a8d32ba36377e3a9c44dab68bc00af74858bb2cebaa8fdae8bf"} Nov 03 20:18:23.896527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:23.894560 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" podStartSLOduration=3.894539503 podStartE2EDuration="3.894539503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:23.888370761 +0000 UTC m=+1013.418716434" watchObservedRunningTime="2026-11-03 20:18:23.894539503 +0000 UTC m=+1013.424885173" Nov 03 20:18:24.728529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:24.727633 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:31.731527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.725950 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:31.830606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.830576 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:31.886531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.882132 5815 scope.go:122] "RemoveContainer" containerID="5ec0fa2b77fbea08fecbee3f93744dcae456d7fc930a66b9709ce64a1dbaff07" Nov 03 20:18:31.902539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.901847 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x"] Nov 03 20:18:31.916535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.908891 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8"] Nov 03 20:18:31.916535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.909800 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.930264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.929692 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x"] Nov 03 20:18:31.930264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.929799 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:31.934092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.934074 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8"] Nov 03 20:18:31.949030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.949002 5815 scope.go:122] "RemoveContainer" containerID="3e7075029dfa4dd4d8f5260aca4be5a1fc8e0c1f802b6d153c2007140ed3fcb4" Nov 03 20:18:31.966225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.966197 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.970502 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.970477 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.970696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.970677 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf6n2\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.970898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.970881 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.971059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.971019 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.971236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.971221 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.971366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.971352 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.971548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.971531 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.973085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.971863 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:31.973577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.973501 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:18:31.999864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:31.999025 5815 scope.go:122] "RemoveContainer" containerID="5f355acf8b142a7368c040aaeb56191ed30d442437c019523aaaa86ab3cea935" Nov 03 20:18:32.037702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.037595 5815 scope.go:122] "RemoveContainer" containerID="0c7319a47e94108eab0fa97c4ff1e6c1ed906f47753f3333fbfdf9af1cd3c21e" Nov 03 20:18:32.050563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.050540 5815 scope.go:122] "RemoveContainer" containerID="beb22d978f61065f3ea6bc216fba112c56fe3991bd59178e1f3cc554e1400299" Nov 03 20:18:32.074580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074383 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.074580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074431 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqd8\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.074580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074471 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.074580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074501 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.074983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074582 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.074983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074605 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.074983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074676 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.074983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.074721 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075413 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075609 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075644 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf6n2\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075692 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075721 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075747 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075798 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075871 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075909 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.075974 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.076017 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.076707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.076698 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.077267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.076759 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.077267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.077192 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.077663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.077611 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.085616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.079371 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.085616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.079680 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.088884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.088780 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf6n2\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.089141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.089101 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.179839 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.179910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180049 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180134 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqd8\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180169 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180196 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180254 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180283 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180308 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.180826 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.181064 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.182533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.181689 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.183329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.183022 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.205530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.185766 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.206928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.206263 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.220531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.217119 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.220531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.217893 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqd8\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.220531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.218651 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.265696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.265627 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:32.293945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.293917 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:32.576184 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:18:32.576153 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076ac19f_9655_4dcc_9b0f_320575410efb.slice/crio-c6abaaafe8e508eb2dc8851912bfafe2bd3e8189068a3825b8e774d1503cb162 WatchSource:0}: Error finding container c6abaaafe8e508eb2dc8851912bfafe2bd3e8189068a3825b8e774d1503cb162: Status 404 returned error can't find the container with id c6abaaafe8e508eb2dc8851912bfafe2bd3e8189068a3825b8e774d1503cb162 Nov 03 20:18:32.579034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.578998 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:32.579136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.579066 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:32.579136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.579095 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:32.589361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.589341 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x"] Nov 03 20:18:32.622069 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:18:32.622041 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d1e076_963f_4c22_b3ef_4ada1063682a.slice/crio-2270d4cb4fbfc2c82173fa49374b32bfe6b3d8dc9e38939bc547594e0dd327d6 WatchSource:0}: Error finding container 2270d4cb4fbfc2c82173fa49374b32bfe6b3d8dc9e38939bc547594e0dd327d6: Status 404 returned error can't find the container with id 2270d4cb4fbfc2c82173fa49374b32bfe6b3d8dc9e38939bc547594e0dd327d6 Nov 03 20:18:32.623156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.623133 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8"] Nov 03 20:18:32.625127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.625095 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:32.625222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.625160 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:32.625222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.625188 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:32.965989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.965880 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" event={"ID":"076ac19f-9655-4dcc-9b0f-320575410efb","Type":"ContainerStarted","Data":"df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d"} Nov 03 20:18:32.965989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.965948 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" event={"ID":"076ac19f-9655-4dcc-9b0f-320575410efb","Type":"ContainerStarted","Data":"c6abaaafe8e508eb2dc8851912bfafe2bd3e8189068a3825b8e774d1503cb162"} Nov 03 20:18:32.977201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.977158 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" event={"ID":"b5d1e076-963f-4c22-b3ef-4ada1063682a","Type":"ContainerStarted","Data":"ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4"} Nov 03 20:18:32.977357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.977267 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" event={"ID":"b5d1e076-963f-4c22-b3ef-4ada1063682a","Type":"ContainerStarted","Data":"2270d4cb4fbfc2c82173fa49374b32bfe6b3d8dc9e38939bc547594e0dd327d6"} Nov 03 20:18:32.997429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:32.997370 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" podStartSLOduration=1.997356565 podStartE2EDuration="1.997356565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:32.991417748 +0000 UTC m=+1022.521763421" watchObservedRunningTime="2026-11-03 20:18:32.997356565 +0000 UTC m=+1022.527702236" Nov 03 20:18:33.025903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.025856 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" podStartSLOduration=2.025843267 podStartE2EDuration="2.025843267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:33.021339523 +0000 UTC m=+1022.551685189" watchObservedRunningTime="2026-11-03 20:18:33.025843267 +0000 UTC m=+1022.556189001" Nov 03 20:18:33.270911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.270815 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:33.276562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.275492 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 20:18:33.276562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.275608 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 20:18:33.312745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.312707 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:33.329227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.328682 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:18:33.329227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:33.328744 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:18:34.153055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.153024 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8"] Nov 03 20:18:34.160615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.160588 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x"] Nov 03 20:18:34.186713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.186677 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5"] Nov 03 20:18:34.187216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.187185 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" containerID="cri-o://5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887" gracePeriod=30 Nov 03 20:18:34.300554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.300188 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:18:34.300554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.300247 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:18:34.409498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.409285 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 20:18:34.426672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:34.420600 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 20:18:35.039736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:35.039707 5815 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" secret="" err="secret \"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\" not found" Nov 03 20:18:35.040075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:35.040058 5815 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" secret="" err="secret \"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\" not found" Nov 03 20:18:35.269532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:35.268811 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 03 20:18:35.269532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:35.268868 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 03 20:18:35.297109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:35.297025 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:18:35.297367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:35.297339 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:18:36.050400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:36.050365 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" containerID="cri-o://df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d" gracePeriod=30 Nov 03 20:18:36.050765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:36.050743 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" containerID="cri-o://ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4" gracePeriod=30 Nov 03 20:18:39.125917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:39.125831 5815 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:18:39.126427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:39.126049 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:18:40.138117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.138091 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:40.140522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.140475 5815 generic.go:356] "Generic (PLEG): container finished" podID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerID="5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887" exitCode=0 Nov 03 20:18:40.140646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.140590 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" event={"ID":"e16ffb8c-97a0-4696-8a83-80d14a4ebab8","Type":"ContainerDied","Data":"5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887"} Nov 03 20:18:40.140646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.140603 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" Nov 03 20:18:40.140646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.140642 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5" event={"ID":"e16ffb8c-97a0-4696-8a83-80d14a4ebab8","Type":"ContainerDied","Data":"60a577176a65376d5b5114b0efcf1bce63f225b99fc521946c71a74063ba9a76"} Nov 03 20:18:40.140791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.140673 5815 scope.go:122] "RemoveContainer" containerID="5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887" Nov 03 20:18:40.153102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.153081 5815 scope.go:122] "RemoveContainer" containerID="5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887" Nov 03 20:18:40.153379 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:18:40.153359 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887\": container with ID starting with 5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887 not found: ID does not exist" containerID="5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887" Nov 03 20:18:40.153486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.153385 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887"} err="failed to get container status \"5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887\": rpc error: code = NotFound desc = could not find container \"5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887\": container with ID starting with 5b3add2d856efd49048a4bca5f97bc74a6ad48044f5e6bb2b81b67a9d0356887 not found: ID does not exist" Nov 03 20:18:40.233709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.233657 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.233978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.233946 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:40.234067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234048 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.234120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234093 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.234184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234171 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.234285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234264 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.234491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234471 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.234581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234535 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.234581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234571 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.235833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.234618 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token\") pod \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\" (UID: \"e16ffb8c-97a0-4696-8a83-80d14a4ebab8\") " Nov 03 20:18:40.235833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.235309 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.236629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.236483 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:40.236873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.236850 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:40.237030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.237009 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:40.237093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.237072 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:40.238617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.238588 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:40.239483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.239447 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:40.239655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.239589 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "kube-api-access-prxsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:40.239788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.239764 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy" pod "e16ffb8c-97a0-4696-8a83-80d14a4ebab8" (UID: "e16ffb8c-97a0-4696-8a83-80d14a4ebab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:40.336293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336265 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336291 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336300 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336309 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336318 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336326 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336334 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.336449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.336342 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxsw\" (UniqueName: \"kubernetes.io/projected/e16ffb8c-97a0-4696-8a83-80d14a4ebab8-kube-api-access-prxsw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:40.474421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.474389 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5"] Nov 03 20:18:40.478944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.478922 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-d2tp5"] Nov 03 20:18:40.600045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:40.600006 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" path="/var/lib/kubelet/pods/e16ffb8c-97a0-4696-8a83-80d14a4ebab8/volumes" Nov 03 20:18:43.860992 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:18:43.860957 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:18:44.518148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.518115 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl"] Nov 03 20:18:44.519323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.519295 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" Nov 03 20:18:44.519323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.519325 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" Nov 03 20:18:44.519486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.519470 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16ffb8c-97a0-4696-8a83-80d14a4ebab8" containerName="istio-proxy" Nov 03 20:18:44.523820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.523800 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.533731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.533695 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl"] Nov 03 20:18:44.718108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718067 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718243 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718287 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718363 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718498 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97g4p\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718565 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718601 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718618 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.718966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.718828 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821209 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97g4p\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821302 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821355 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821393 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821500 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821585 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821719 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821762 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.821806 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.822296 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.823008 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.823302 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.823582 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.827084 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.828624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.828034 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.838762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.838736 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.853195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.853164 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97g4p\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.854695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.854665 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:44.910842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.910789 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:44.921928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:44.921909 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:45.023681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023645 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.023866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023705 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.023866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023741 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.023866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023765 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.023866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023819 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023867 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.024084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023918 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.024084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023947 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.024084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.023978 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024055 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024083 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024138 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024167 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024179 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024203 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024232 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024298 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.024687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024360 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token\") pod \"076ac19f-9655-4dcc-9b0f-320575410efb\" (UID: \"076ac19f-9655-4dcc-9b0f-320575410efb\") " Nov 03 20:18:45.024687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024378 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.024687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024395 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token\") pod \"b5d1e076-963f-4c22-b3ef-4ada1063682a\" (UID: \"b5d1e076-963f-4c22-b3ef-4ada1063682a\") " Nov 03 20:18:45.025248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024879 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:45.025248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.024996 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.025444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.025317 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.025444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.025339 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.025444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.025353 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.025444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.025367 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076ac19f-9655-4dcc-9b0f-320575410efb-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.025747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.025646 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.025959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.025935 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:45.026174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026153 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.026443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026418 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.026671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026646 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.026797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026775 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "kube-api-access-tcqd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:45.026797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026786 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.026934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026817 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "kube-api-access-gf6n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:45.026934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.026907 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:45.027195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.027176 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.027381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.027355 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:45.027538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.027485 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:45.027963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.027946 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo" pod "b5d1e076-963f-4c22-b3ef-4ada1063682a" (UID: "b5d1e076-963f-4c22-b3ef-4ada1063682a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:45.028604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.028588 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token" pod "076ac19f-9655-4dcc-9b0f-320575410efb" (UID: "076ac19f-9655-4dcc-9b0f-320575410efb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:45.127257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.127092 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d1e076-963f-4c22-b3ef-4ada1063682a-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.127257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.127155 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.127257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.127171 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.127861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.127800 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.127861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.127829 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.127843 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128071 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128285 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128310 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076ac19f-9655-4dcc-9b0f-320575410efb-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128426 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqd8\" (UniqueName: \"kubernetes.io/projected/b5d1e076-963f-4c22-b3ef-4ada1063682a-kube-api-access-tcqd8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128451 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf6n2\" (UniqueName: \"kubernetes.io/projected/076ac19f-9655-4dcc-9b0f-320575410efb-kube-api-access-gf6n2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128466 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076ac19f-9655-4dcc-9b0f-320575410efb-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128479 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.128949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.128688 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d1e076-963f-4c22-b3ef-4ada1063682a-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:45.148008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.147980 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:45.190233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.190203 5815 generic.go:356] "Generic (PLEG): container finished" podID="076ac19f-9655-4dcc-9b0f-320575410efb" containerID="df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d" exitCode=0 Nov 03 20:18:45.190401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.190263 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" event={"ID":"076ac19f-9655-4dcc-9b0f-320575410efb","Type":"ContainerDied","Data":"df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d"} Nov 03 20:18:45.190401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.190369 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" event={"ID":"076ac19f-9655-4dcc-9b0f-320575410efb","Type":"ContainerDied","Data":"c6abaaafe8e508eb2dc8851912bfafe2bd3e8189068a3825b8e774d1503cb162"} Nov 03 20:18:45.190547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.190275 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x" Nov 03 20:18:45.190547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.190404 5815 scope.go:122] "RemoveContainer" containerID="df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d" Nov 03 20:18:45.199672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.199651 5815 generic.go:356] "Generic (PLEG): container finished" podID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerID="ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4" exitCode=0 Nov 03 20:18:45.199886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.199706 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" Nov 03 20:18:45.200982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.199729 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" event={"ID":"b5d1e076-963f-4c22-b3ef-4ada1063682a","Type":"ContainerDied","Data":"ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4"} Nov 03 20:18:45.200982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.200859 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8" event={"ID":"b5d1e076-963f-4c22-b3ef-4ada1063682a","Type":"ContainerDied","Data":"2270d4cb4fbfc2c82173fa49374b32bfe6b3d8dc9e38939bc547594e0dd327d6"} Nov 03 20:18:45.216721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.215803 5815 scope.go:122] "RemoveContainer" containerID="df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d" Nov 03 20:18:45.216721 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:18:45.216163 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d\": container with ID starting with df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d not found: ID does not exist" containerID="df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d" Nov 03 20:18:45.216721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.216202 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d"} err="failed to get container status \"df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d\": rpc error: code = NotFound desc = could not find container \"df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d\": container with ID starting with df72f55dcfcf4fa9ad61d8a1cef390430667393666f2d37d920e496df9276c8d not found: ID does not exist" Nov 03 20:18:45.216721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.216224 5815 scope.go:122] "RemoveContainer" containerID="ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4" Nov 03 20:18:45.224226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.224203 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x"] Nov 03 20:18:45.234624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.234597 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-8gb9x"] Nov 03 20:18:45.256235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.256201 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8"] Nov 03 20:18:45.264331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.264217 5815 scope.go:122] "RemoveContainer" containerID="ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4" Nov 03 20:18:45.265083 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:18:45.264982 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4\": container with ID starting with ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4 not found: ID does not exist" containerID="ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4" Nov 03 20:18:45.265083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.265018 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4"} err="failed to get container status \"ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4\": rpc error: code = NotFound desc = could not find container \"ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4\": container with ID starting with ccb0e36429ca8dc17caeb8bc997b798ef8244dffdc21393e4f18bc5a3b2eddf4 not found: ID does not exist" Nov 03 20:18:45.266159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.266144 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-fzdh8"] Nov 03 20:18:45.451415 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:18:45.451390 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7fb429_124e_4008_894f_fc3262dc9682.slice/crio-8b264f5b7f6ea7a3af5c0071b2b8cc06afad88c40250d1ad362ec2ad525e2316 WatchSource:0}: Error finding container 8b264f5b7f6ea7a3af5c0071b2b8cc06afad88c40250d1ad362ec2ad525e2316: Status 404 returned error can't find the container with id 8b264f5b7f6ea7a3af5c0071b2b8cc06afad88c40250d1ad362ec2ad525e2316 Nov 03 20:18:45.453102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.453083 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl"] Nov 03 20:18:45.454210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.454176 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:45.454265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.454246 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:45.454300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:45.454274 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:18:46.214411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:46.214374 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" event={"ID":"6e7fb429-124e-4008-894f-fc3262dc9682","Type":"ContainerStarted","Data":"017a2bff9c8abe1eaa05856f4fd549dab0346f7a76f5b7e33ca54b4480e0d69e"} Nov 03 20:18:46.214868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:46.214436 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" event={"ID":"6e7fb429-124e-4008-894f-fc3262dc9682","Type":"ContainerStarted","Data":"8b264f5b7f6ea7a3af5c0071b2b8cc06afad88c40250d1ad362ec2ad525e2316"} Nov 03 20:18:46.246531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:46.243680 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podStartSLOduration=2.2436605 podStartE2EDuration="2.2436605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:46.241718236 +0000 UTC m=+1035.772063909" watchObservedRunningTime="2026-11-03 20:18:46.2436605 +0000 UTC m=+1035.774006173" Nov 03 20:18:46.605079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:46.604994 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" path="/var/lib/kubelet/pods/076ac19f-9655-4dcc-9b0f-320575410efb/volumes" Nov 03 20:18:46.605651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:46.605628 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" path="/var/lib/kubelet/pods/b5d1e076-963f-4c22-b3ef-4ada1063682a/volumes" Nov 03 20:18:47.148435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:47.148401 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:47.150179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:47.150142 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:47.150315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:47.150206 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:48.148334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:48.148296 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:48.148784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:48.148356 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:49.148995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:49.148964 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:49.149422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:49.149027 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:49.200343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:49.200230 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:18:50.119402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.119362 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp"] Nov 03 20:18:50.119997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.119979 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" Nov 03 20:18:50.119997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.119996 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" Nov 03 20:18:50.120164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.120024 5815 cpu_manager.go:414] "containerMap: removing container" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" Nov 03 20:18:50.120164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.120030 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" Nov 03 20:18:50.120164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.120114 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="076ac19f-9655-4dcc-9b0f-320575410efb" containerName="istio-proxy" Nov 03 20:18:50.120164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.120126 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d1e076-963f-4c22-b3ef-4ada1063682a" containerName="istio-proxy" Nov 03 20:18:50.125264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.125245 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.134558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.133087 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:18:50.134558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.133370 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:18:50.135037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.134812 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp"] Nov 03 20:18:50.148876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.148851 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:50.148988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.148900 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:50.297838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.297792 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.298339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.297870 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.298339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.297958 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txx6h\" (UniqueName: \"kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.418340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.418250 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.418501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.418354 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.418501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.418454 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txx6h\" (UniqueName: \"kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.419013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.418990 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.422913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.422887 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.433786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.433764 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txx6h\" (UniqueName: \"kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h\") pod \"maas-api-revoke-sub-6610ac02-ck5qp\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:50.446284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:50.446258 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:51.152839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:51.151952 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:51.152839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:51.152063 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:51.216619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:51.215956 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp"] Nov 03 20:18:51.334527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:51.334474 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" event={"ID":"c627dbc1-8631-42e5-8b02-9898d2f62ca4","Type":"ContainerStarted","Data":"ac09187fdaf918f7cdab10688f6b7adf9b4f4f854c08a57cc5e60cae49628349"} Nov 03 20:18:52.149143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:52.149108 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:52.149341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:52.149162 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:52.445857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:52.445652 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" event={"ID":"c627dbc1-8631-42e5-8b02-9898d2f62ca4","Type":"ContainerStarted","Data":"ef2f1aa0656144c6009f1ce01fb3126b1effe1d5f8986ff2809787f59a12de79"} Nov 03 20:18:52.501535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:52.501321 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" podStartSLOduration=2.501299912 podStartE2EDuration="2.501299912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:52.463892637 +0000 UTC m=+1041.994238321" watchObservedRunningTime="2026-11-03 20:18:52.501299912 +0000 UTC m=+1042.031645585" Nov 03 20:18:53.149537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:53.148889 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 20:18:53.149537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:53.148951 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 20:18:53.457957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:53.457856 5815 generic.go:356] "Generic (PLEG): container finished" podID="c627dbc1-8631-42e5-8b02-9898d2f62ca4" containerID="ef2f1aa0656144c6009f1ce01fb3126b1effe1d5f8986ff2809787f59a12de79" exitCode=0 Nov 03 20:18:53.458357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:53.457944 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" event={"ID":"c627dbc1-8631-42e5-8b02-9898d2f62ca4","Type":"ContainerDied","Data":"ef2f1aa0656144c6009f1ce01fb3126b1effe1d5f8986ff2809787f59a12de79"} Nov 03 20:18:54.153309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.153280 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:54.153788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.153770 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:54.154460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.154444 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:18:54.840751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.840722 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:54.920151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.920112 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h\" (UniqueName: \"kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h\") pod \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " Nov 03 20:18:54.920353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.920217 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca\") pod \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " Nov 03 20:18:54.920353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.920292 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca\") pod \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\" (UID: \"c627dbc1-8631-42e5-8b02-9898d2f62ca4\") " Nov 03 20:18:54.920691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.920648 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca" pod "c627dbc1-8631-42e5-8b02-9898d2f62ca4" (UID: "c627dbc1-8631-42e5-8b02-9898d2f62ca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:54.920904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.920879 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:54.922699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.922672 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca" pod "c627dbc1-8631-42e5-8b02-9898d2f62ca4" (UID: "c627dbc1-8631-42e5-8b02-9898d2f62ca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:18:54.922779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:54.922715 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h" pod "c627dbc1-8631-42e5-8b02-9898d2f62ca4" (UID: "c627dbc1-8631-42e5-8b02-9898d2f62ca4"). InnerVolumeSpecName "kube-api-access-txx6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:55.021891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:55.021860 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txx6h\" (UniqueName: \"kubernetes.io/projected/c627dbc1-8631-42e5-8b02-9898d2f62ca4-kube-api-access-txx6h\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:55.021891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:55.021889 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c627dbc1-8631-42e5-8b02-9898d2f62ca4-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:55.521476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:55.521435 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" Nov 03 20:18:55.521697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:55.521483 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp" event={"ID":"c627dbc1-8631-42e5-8b02-9898d2f62ca4","Type":"ContainerDied","Data":"ac09187fdaf918f7cdab10688f6b7adf9b4f4f854c08a57cc5e60cae49628349"} Nov 03 20:18:55.521697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:55.521566 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac09187fdaf918f7cdab10688f6b7adf9b4f4f854c08a57cc5e60cae49628349" Nov 03 20:18:56.050956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:56.050893 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm"] Nov 03 20:18:56.055747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:56.055722 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60164d7c-w4ppm"] Nov 03 20:18:56.615674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:56.615636 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5388799-7784-418d-9e0b-eaf69c8f2bb8" path="/var/lib/kubelet/pods/a5388799-7784-418d-9e0b-eaf69c8f2bb8/volumes" Nov 03 20:18:57.668116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.668028 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8"] Nov 03 20:18:57.668672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.668655 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c627dbc1-8631-42e5-8b02-9898d2f62ca4" containerName="revoke-keys" Nov 03 20:18:57.668672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.668671 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c627dbc1-8631-42e5-8b02-9898d2f62ca4" containerName="revoke-keys" Nov 03 20:18:57.668793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.668761 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c627dbc1-8631-42e5-8b02-9898d2f62ca4" containerName="revoke-keys" Nov 03 20:18:57.673753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.673733 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.679869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.679727 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:18:57.679869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.679840 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:18:57.686528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.686480 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8"] Nov 03 20:18:57.767988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.767955 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.768146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.768008 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljt4q\" (UniqueName: \"kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.768265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.768204 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.868981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.868943 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.869133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.868995 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljt4q\" (UniqueName: \"kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.869217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.869134 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.869853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.869808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.871736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.871710 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:57.882362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:57.882332 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljt4q\" (UniqueName: \"kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q\") pod \"maas-api-revoke-sub-bd2f6661-82fh8\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:58.006539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:58.000738 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:18:58.453952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:58.453926 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8"] Nov 03 20:18:58.456378 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:18:58.456351 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe529a73_e2be_4675_93f6_f5aa0fa3fe64.slice/crio-e5f0dea9e231d6a8d6b6cc546b5963e0c227217a71b562893882485dd7554868 WatchSource:0}: Error finding container e5f0dea9e231d6a8d6b6cc546b5963e0c227217a71b562893882485dd7554868: Status 404 returned error can't find the container with id e5f0dea9e231d6a8d6b6cc546b5963e0c227217a71b562893882485dd7554868 Nov 03 20:18:58.549755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:58.549723 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" event={"ID":"fe529a73-e2be-4675-93f6-f5aa0fa3fe64","Type":"ContainerStarted","Data":"e5f0dea9e231d6a8d6b6cc546b5963e0c227217a71b562893882485dd7554868"} Nov 03 20:18:59.535991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.535961 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz"] Nov 03 20:18:59.543874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.543842 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.569473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.568190 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz"] Nov 03 20:18:59.574018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.573976 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" event={"ID":"fe529a73-e2be-4675-93f6-f5aa0fa3fe64","Type":"ContainerStarted","Data":"5290f688bca335726cfd53640a450a361111dc66d862c1d4aa1f0fc710e1847b"} Nov 03 20:18:59.596395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.596341 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" podStartSLOduration=2.596322254 podStartE2EDuration="2.596322254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:59.591612252 +0000 UTC m=+1049.121957951" watchObservedRunningTime="2026-11-03 20:18:59.596322254 +0000 UTC m=+1049.126667927" Nov 03 20:18:59.628740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.628698 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.628936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.628918 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wqp\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629079 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629224 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629332 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629383 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629409 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629536 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.629650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.629625 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730327 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wqp\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730430 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730496 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730591 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730649 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730681 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730756 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.730836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.730787 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.731128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.731081 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.731249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.731178 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.731367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.731345 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.731641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.731583 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.731753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.731671 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.733581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.733543 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.733838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.733790 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.742723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.742705 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.742808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.742793 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wqp\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:18:59.873785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:18:59.873703 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:19:00.171947 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:19:00.171905 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd5c516_be9e_49f4_a2c5_4cb3e92af15a.slice/crio-d2ea7d723e1f2dc164036e475ff77d025d67292bfe6d902b078ccfcdf05e5f33 WatchSource:0}: Error finding container d2ea7d723e1f2dc164036e475ff77d025d67292bfe6d902b078ccfcdf05e5f33: Status 404 returned error can't find the container with id d2ea7d723e1f2dc164036e475ff77d025d67292bfe6d902b078ccfcdf05e5f33 Nov 03 20:19:00.172870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.172846 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz"] Nov 03 20:19:00.175300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.175262 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:00.175417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.175332 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:00.175417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.175367 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:00.596713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.596680 5815 generic.go:356] "Generic (PLEG): container finished" podID="fe529a73-e2be-4675-93f6-f5aa0fa3fe64" containerID="5290f688bca335726cfd53640a450a361111dc66d862c1d4aa1f0fc710e1847b" exitCode=0 Nov 03 20:19:00.597166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.596770 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" event={"ID":"fe529a73-e2be-4675-93f6-f5aa0fa3fe64","Type":"ContainerDied","Data":"5290f688bca335726cfd53640a450a361111dc66d862c1d4aa1f0fc710e1847b"} Nov 03 20:19:00.598906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.598882 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" event={"ID":"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a","Type":"ContainerStarted","Data":"16dc4aa6a187b3af9d7e99c1bc86a58367fe4235712268203764e64f19aa9b58"} Nov 03 20:19:00.599114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.599096 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" event={"ID":"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a","Type":"ContainerStarted","Data":"d2ea7d723e1f2dc164036e475ff77d025d67292bfe6d902b078ccfcdf05e5f33"} Nov 03 20:19:00.654061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.654014 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podStartSLOduration=1.653996646 podStartE2EDuration="1.653996646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:00.652322842 +0000 UTC m=+1050.182668515" watchObservedRunningTime="2026-11-03 20:19:00.653996646 +0000 UTC m=+1050.184342319" Nov 03 20:19:00.877717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:00.877621 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:19:01.612530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.612413 5815 generic.go:356] "Generic (PLEG): container finished" podID="46128744-78a6-4cef-b813-11b4202b3aab" containerID="00e356df1ee7e5a33010c4d7cc375e580a87d437468ed3f3bf75c653e7b8e353" exitCode=0 Nov 03 20:19:01.612530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.612487 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" event={"ID":"46128744-78a6-4cef-b813-11b4202b3aab","Type":"ContainerDied","Data":"00e356df1ee7e5a33010c4d7cc375e580a87d437468ed3f3bf75c653e7b8e353"} Nov 03 20:19:01.613603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.613154 5815 scope.go:122] "RemoveContainer" containerID="00e356df1ee7e5a33010c4d7cc375e580a87d437468ed3f3bf75c653e7b8e353" Nov 03 20:19:01.614948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.614910 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:01.615070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.615003 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:01.615070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.615045 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:01.907563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.902231 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:19:01.907563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.902293 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 20:19:01.997402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:01.990670 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:19:02.120756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.120694 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q\" (UniqueName: \"kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q\") pod \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " Nov 03 20:19:02.120955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.120821 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca\") pod \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " Nov 03 20:19:02.121036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.120953 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca\") pod \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\" (UID: \"fe529a73-e2be-4675-93f6-f5aa0fa3fe64\") " Nov 03 20:19:02.121255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.121224 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca" pod "fe529a73-e2be-4675-93f6-f5aa0fa3fe64" (UID: "fe529a73-e2be-4675-93f6-f5aa0fa3fe64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:02.121696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.121668 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:02.130837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.129981 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q" pod "fe529a73-e2be-4675-93f6-f5aa0fa3fe64" (UID: "fe529a73-e2be-4675-93f6-f5aa0fa3fe64"). InnerVolumeSpecName "kube-api-access-ljt4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:02.130837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.130361 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca" pod "fe529a73-e2be-4675-93f6-f5aa0fa3fe64" (UID: "fe529a73-e2be-4675-93f6-f5aa0fa3fe64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:02.223191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.223161 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljt4q\" (UniqueName: \"kubernetes.io/projected/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-kube-api-access-ljt4q\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:02.223191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.223190 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe529a73-e2be-4675-93f6-f5aa0fa3fe64-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:02.631330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.631289 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" event={"ID":"46128744-78a6-4cef-b813-11b4202b3aab","Type":"ContainerStarted","Data":"b869f549d3b076f596208fa58bd8d0237449632e733d71a9c573709c6765a8fa"} Nov 03 20:19:02.633792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.633714 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" event={"ID":"fe529a73-e2be-4675-93f6-f5aa0fa3fe64","Type":"ContainerDied","Data":"e5f0dea9e231d6a8d6b6cc546b5963e0c227217a71b562893882485dd7554868"} Nov 03 20:19:02.633792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.633763 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5f0dea9e231d6a8d6b6cc546b5963e0c227217a71b562893882485dd7554868" Nov 03 20:19:02.633792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.633783 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8" Nov 03 20:19:02.881591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.878887 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:19:02.881591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:02.878950 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 20:19:03.079528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.075866 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x"] Nov 03 20:19:03.159612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.155688 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1438dd0d-mpz5x"] Nov 03 20:19:03.551778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.551744 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:19:03.551946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.551929 5815 patch_prober.go:29] interesting pod/e2e-api-a-98148a-openshift-default-fbb48db79-cd564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:19:03.552033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.551965 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:19:03.875256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.874857 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:19:03.875256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:03.874920 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 20:19:04.562355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:04.561674 5815 patch_prober.go:29] interesting pod/e2e-api-a-98148a-openshift-default-fbb48db79-cd564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:19:04.562355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:04.561776 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:19:04.629031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:04.628996 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807ac60d-de18-4ab4-bd5d-27e64229f29e" path="/var/lib/kubelet/pods/807ac60d-de18-4ab4-bd5d-27e64229f29e/volumes" Nov 03 20:19:04.877578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:04.877465 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:19:04.878194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:04.878161 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 20:19:05.552473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:05.552438 5815 patch_prober.go:29] interesting pod/e2e-api-a-98148a-openshift-default-fbb48db79-cd564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:19:05.552705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:05.552490 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:19:05.881996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:05.881900 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:19:05.882654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:05.882620 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 20:19:06.053958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.053888 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm"] Nov 03 20:19:06.095674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.094712 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01085c78-rlglm"] Nov 03 20:19:06.551966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.551931 5815 patch_prober.go:29] interesting pod/e2e-api-a-98148a-openshift-default-fbb48db79-cd564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:19:06.552154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.551994 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:19:06.609903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.609863 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f85b59-de69-4aed-b299-f35795ff644a" path="/var/lib/kubelet/pods/d7f85b59-de69-4aed-b299-f35795ff644a/volumes" Nov 03 20:19:06.884003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.883907 5815 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Nov 03 20:19:06.884639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:06.884609 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Nov 03 20:19:07.025518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:07.025471 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8"] Nov 03 20:19:07.030147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:07.030120 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ef4153-bcvq8"] Nov 03 20:19:07.553032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:07.551869 5815 patch_prober.go:29] interesting pod/e2e-api-a-98148a-openshift-default-fbb48db79-cd564 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:19:07.553032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:07.551929 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 20:19:08.521019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.520993 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:19:08.522060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.522038 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:19:08.522977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.522947 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 20:19:08.600177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.598584 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714e3d1d-490f-40b4-ac26-0af71e16daa4" path="/var/lib/kubelet/pods/714e3d1d-490f-40b4-ac26-0af71e16daa4/volumes" Nov 03 20:19:08.857491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.857411 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:19:08.858038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.858006 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:19:08.858789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:08.858770 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:19:09.027749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:09.027714 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s"] Nov 03 20:19:09.032314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:09.032285 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae855749-s5f9s"] Nov 03 20:19:10.587141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:10.587112 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a" path="/var/lib/kubelet/pods/2b6d7ce9-e29b-494f-9e1c-fbeb80e9cc4a/volumes" Nov 03 20:19:13.853720 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:19:13.853680 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:19:15.108777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:15.108553 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p"] Nov 03 20:19:15.156889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:15.156843 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd027bf1-z984p"] Nov 03 20:19:16.647535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:16.644930 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b692a65-f0f3-424a-bae6-e8ed8e3df112" path="/var/lib/kubelet/pods/2b692a65-f0f3-424a-bae6-e8ed8e3df112/volumes" Nov 03 20:19:17.274534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:17.272675 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:19:17.274534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:17.272784 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:17.770983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:17.770946 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564"] Nov 03 20:19:17.790489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:17.790446 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" containerID="cri-o://b869f549d3b076f596208fa58bd8d0237449632e733d71a9c573709c6765a8fa" gracePeriod=30 Nov 03 20:19:20.028804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:20.028770 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq"] Nov 03 20:19:20.033706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:20.033674 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a28c5d52-x8fzq"] Nov 03 20:19:20.588643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:20.588612 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900dc148-7955-4626-af6d-3a0ada8f6298" path="/var/lib/kubelet/pods/900dc148-7955-4626-af6d-3a0ada8f6298/volumes" Nov 03 20:19:23.091658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.091158 5815 generic.go:356] "Generic (PLEG): container finished" podID="46128744-78a6-4cef-b813-11b4202b3aab" containerID="b869f549d3b076f596208fa58bd8d0237449632e733d71a9c573709c6765a8fa" exitCode=0 Nov 03 20:19:23.091658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.091437 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" event={"ID":"46128744-78a6-4cef-b813-11b4202b3aab","Type":"ContainerDied","Data":"b869f549d3b076f596208fa58bd8d0237449632e733d71a9c573709c6765a8fa"} Nov 03 20:19:23.091658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.091498 5815 scope.go:122] "RemoveContainer" containerID="00e356df1ee7e5a33010c4d7cc375e580a87d437468ed3f3bf75c653e7b8e353" Nov 03 20:19:23.197042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.197013 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:19:23.291724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.291688 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.291896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.291751 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.291896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.291807 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.291999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.291912 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.292064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292048 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.292124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292060 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:23.292124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292096 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:23.292124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292107 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.292292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292147 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.292292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292219 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.292400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292311 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy\") pod \"46128744-78a6-4cef-b813-11b4202b3aab\" (UID: \"46128744-78a6-4cef-b813-11b4202b3aab\") " Nov 03 20:19:23.292400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292341 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:23.292719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292690 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:23.292893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292789 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:23.292893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292860 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.292893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292880 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.293242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292895 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.293242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.292912 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46128744-78a6-4cef-b813-11b4202b3aab-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.294760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.294727 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:23.294760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.294749 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:19:23.295060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.295039 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "kube-api-access-l9kxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:23.295311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.295283 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo" pod "46128744-78a6-4cef-b813-11b4202b3aab" (UID: "46128744-78a6-4cef-b813-11b4202b3aab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:19:23.394761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.394679 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46128744-78a6-4cef-b813-11b4202b3aab-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.394761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.394708 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.394761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.394722 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.394761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.394734 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46128744-78a6-4cef-b813-11b4202b3aab-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:23.394761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:23.394748 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9kxw\" (UniqueName: \"kubernetes.io/projected/46128744-78a6-4cef-b813-11b4202b3aab-kube-api-access-l9kxw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:24.127554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:24.125053 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" event={"ID":"46128744-78a6-4cef-b813-11b4202b3aab","Type":"ContainerDied","Data":"70e5915b9c1f2c2fe6172e48d2f621947fe006906cc4ecfa922b458a979baf00"} Nov 03 20:19:24.127554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:24.125129 5815 scope.go:122] "RemoveContainer" containerID="b869f549d3b076f596208fa58bd8d0237449632e733d71a9c573709c6765a8fa" Nov 03 20:19:24.127554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:24.125245 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564" Nov 03 20:19:24.163185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:24.163160 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564"] Nov 03 20:19:24.169539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:24.169495 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-98148a-openshift-default-fbb48db79-cd564"] Nov 03 20:19:24.588916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:24.588886 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46128744-78a6-4cef-b813-11b4202b3aab" path="/var/lib/kubelet/pods/46128744-78a6-4cef-b813-11b4202b3aab/volumes" Nov 03 20:19:25.537400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.537364 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7570-ddf4b9"] Nov 03 20:19:25.538175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.538153 5815 cpu_manager.go:414] "containerMap: removing container" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" Nov 03 20:19:25.538278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.538176 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" Nov 03 20:19:25.538278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.538226 5815 cpu_manager.go:414] "containerMap: removing container" podUID="fe529a73-e2be-4675-93f6-f5aa0fa3fe64" containerName="revoke-keys" Nov 03 20:19:25.538278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.538234 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe529a73-e2be-4675-93f6-f5aa0fa3fe64" containerName="revoke-keys" Nov 03 20:19:25.538424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.538398 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" Nov 03 20:19:25.538424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.538421 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe529a73-e2be-4675-93f6-f5aa0fa3fe64" containerName="revoke-keys" Nov 03 20:19:25.543612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.543581 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:25.548496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.548477 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6p5pq\"" Nov 03 20:19:25.548810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.548786 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7570-ddf4b9"] Nov 03 20:19:25.737090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.737054 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66spd\" (UniqueName: \"kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd\") pod \"test-curl-7570-ddf4b9\" (UID: \"3d80b970-f38a-4862-b979-07c932f9f41e\") " pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:25.838073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.837963 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66spd\" (UniqueName: \"kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd\") pod \"test-curl-7570-ddf4b9\" (UID: \"3d80b970-f38a-4862-b979-07c932f9f41e\") " pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:25.851123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.851100 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66spd\" (UniqueName: \"kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd\") pod \"test-curl-7570-ddf4b9\" (UID: \"3d80b970-f38a-4862-b979-07c932f9f41e\") " pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:25.858069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:25.858047 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:26.139997 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:19:26.139964 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d80b970_f38a_4862_b979_07c932f9f41e.slice/crio-f10127de381795bc06532fda9fa7e4980568cca4e782f33dfeebc8f2aecd1d25 WatchSource:0}: Error finding container f10127de381795bc06532fda9fa7e4980568cca4e782f33dfeebc8f2aecd1d25: Status 404 returned error can't find the container with id f10127de381795bc06532fda9fa7e4980568cca4e782f33dfeebc8f2aecd1d25 Nov 03 20:19:26.144042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:26.141254 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7570-ddf4b9"] Nov 03 20:19:26.144807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:26.144780 5815 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:19:27.160713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:27.160673 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7570-ddf4b9" event={"ID":"3d80b970-f38a-4862-b979-07c932f9f41e","Type":"ContainerStarted","Data":"f10127de381795bc06532fda9fa7e4980568cca4e782f33dfeebc8f2aecd1d25"} Nov 03 20:19:30.027038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:30.027000 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v"] Nov 03 20:19:30.033168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:30.033133 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2c0b564-fg75v"] Nov 03 20:19:30.587073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:30.587002 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e6ad83-0421-4b0e-9ab8-654eb386beb0" path="/var/lib/kubelet/pods/65e6ad83-0421-4b0e-9ab8-654eb386beb0/volumes" Nov 03 20:19:31.235674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:31.235637 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7570-ddf4b9" event={"ID":"3d80b970-f38a-4862-b979-07c932f9f41e","Type":"ContainerStarted","Data":"3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9"} Nov 03 20:19:31.300537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:31.300086 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7570-ddf4b9" podStartSLOduration=2.151231912 podStartE2EDuration="6.300062044s" totalImagesPullingTime="4.148830132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:31.275185064 +0000 UTC m=+1080.805530739" watchObservedRunningTime="2026-11-03 20:19:31.300062044 +0000 UTC m=+1080.830407717" Nov 03 20:19:32.084557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.078850 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7570-ddf4b9"] Nov 03 20:19:32.201297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.201261 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7570-ddf4b9"] Nov 03 20:19:32.655247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.655210 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd"] Nov 03 20:19:32.655995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.655974 5815 cpu_manager.go:414] "containerMap: removing container" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" Nov 03 20:19:32.656104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.655997 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" Nov 03 20:19:32.656104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.656082 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" containerName="test-curl-7570-ddf4b9" Nov 03 20:19:32.656104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.656091 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" containerName="test-curl-7570-ddf4b9" Nov 03 20:19:32.656274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.656260 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" containerName="test-curl-7570-ddf4b9" Nov 03 20:19:32.656326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.656277 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="46128744-78a6-4cef-b813-11b4202b3aab" containerName="istio-proxy" Nov 03 20:19:32.661601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.661580 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.665798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.665722 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:19:32.666732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.666711 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:19:32.669703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.669671 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd"] Nov 03 20:19:32.698039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.698017 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.698154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.698083 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.698154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.698119 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngx2\" (UniqueName: \"kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.799178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.799139 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.799345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.799194 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngx2\" (UniqueName: \"kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.799465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.799447 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.800097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.800063 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.801785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.801759 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.811433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.811410 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngx2\" (UniqueName: \"kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2\") pod \"maas-api-revoke-keys-e2e-mt-9b65143e-77qcd\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:32.978783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:32.978751 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:33.258057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:33.257989 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7570-ddf4b9" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" containerName="test-curl-7570-ddf4b9" containerID="cri-o://3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9" gracePeriod=2 Nov 03 20:19:33.276048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:33.275988 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd"] Nov 03 20:19:34.268277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:34.268242 5815 generic.go:356] "Generic (PLEG): container finished" podID="a8d492ac-c387-473a-9c5b-17b652e7455d" containerID="20e3cb6e02cb37c33496154cf6a7df205e59773008a76a71b0a9bce95f056c2d" exitCode=0 Nov 03 20:19:34.268633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:34.268337 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" event={"ID":"a8d492ac-c387-473a-9c5b-17b652e7455d","Type":"ContainerDied","Data":"20e3cb6e02cb37c33496154cf6a7df205e59773008a76a71b0a9bce95f056c2d"} Nov 03 20:19:34.268633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:34.268399 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" event={"ID":"a8d492ac-c387-473a-9c5b-17b652e7455d","Type":"ContainerStarted","Data":"4de461499c9aa02c68c510f5eb4bb343b652a76c163da10af35f1f97913ee06b"} Nov 03 20:19:35.489731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.489706 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:35.537905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.537875 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca\") pod \"a8d492ac-c387-473a-9c5b-17b652e7455d\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " Nov 03 20:19:35.538062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.537963 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2\" (UniqueName: \"kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2\") pod \"a8d492ac-c387-473a-9c5b-17b652e7455d\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " Nov 03 20:19:35.538062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.538003 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca\") pod \"a8d492ac-c387-473a-9c5b-17b652e7455d\" (UID: \"a8d492ac-c387-473a-9c5b-17b652e7455d\") " Nov 03 20:19:35.538736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.538483 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca" pod "a8d492ac-c387-473a-9c5b-17b652e7455d" (UID: "a8d492ac-c387-473a-9c5b-17b652e7455d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:35.538736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.538690 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.540289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.540261 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2" pod "a8d492ac-c387-473a-9c5b-17b652e7455d" (UID: "a8d492ac-c387-473a-9c5b-17b652e7455d"). InnerVolumeSpecName "kube-api-access-8ngx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.540778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.540760 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca" pod "a8d492ac-c387-473a-9c5b-17b652e7455d" (UID: "a8d492ac-c387-473a-9c5b-17b652e7455d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:35.555000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.554978 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:35.650530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.650111 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd\" (UniqueName: \"kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd\") pod \"3d80b970-f38a-4862-b979-07c932f9f41e\" (UID: \"3d80b970-f38a-4862-b979-07c932f9f41e\") " Nov 03 20:19:35.656585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.650895 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d492ac-c387-473a-9c5b-17b652e7455d-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.656585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.650923 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngx2\" (UniqueName: \"kubernetes.io/projected/a8d492ac-c387-473a-9c5b-17b652e7455d-kube-api-access-8ngx2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:35.671716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.665952 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd" pod "3d80b970-f38a-4862-b979-07c932f9f41e" (UID: "3d80b970-f38a-4862-b979-07c932f9f41e"). InnerVolumeSpecName "kube-api-access-66spd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:35.752006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:35.751969 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66spd\" (UniqueName: \"kubernetes.io/projected/3d80b970-f38a-4862-b979-07c932f9f41e-kube-api-access-66spd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:36.287736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.287707 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" Nov 03 20:19:36.287896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.287736 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd" event={"ID":"a8d492ac-c387-473a-9c5b-17b652e7455d","Type":"ContainerDied","Data":"4de461499c9aa02c68c510f5eb4bb343b652a76c163da10af35f1f97913ee06b"} Nov 03 20:19:36.287896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.287797 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de461499c9aa02c68c510f5eb4bb343b652a76c163da10af35f1f97913ee06b" Nov 03 20:19:36.290837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.290808 5815 generic.go:356] "Generic (PLEG): container finished" podID="3d80b970-f38a-4862-b979-07c932f9f41e" containerID="3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9" exitCode=137 Nov 03 20:19:36.290986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.290858 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:36.291120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.290891 5815 scope.go:122] "RemoveContainer" containerID="3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9" Nov 03 20:19:36.303959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.303940 5815 scope.go:122] "RemoveContainer" containerID="3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9" Nov 03 20:19:36.304316 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:19:36.304224 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9\": container with ID starting with 3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9 not found: ID does not exist" containerID="3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9" Nov 03 20:19:36.304316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.304258 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9"} err="failed to get container status \"3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9\": rpc error: code = NotFound desc = could not find container \"3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9\": container with ID starting with 3baf4421deb43a5b12e334d8fca6fee31e50d91e61404c6e58ea993435f596f9 not found: ID does not exist" Nov 03 20:19:36.309978 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:19:36.309896 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7570-ddf4b9\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:36.322327 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:19:36.322289 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7570-ddf4b9\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" pod="opendatahub/test-curl-7570-ddf4b9" Nov 03 20:19:36.587722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:36.587650 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d80b970-f38a-4862-b979-07c932f9f41e" path="/var/lib/kubelet/pods/3d80b970-f38a-4862-b979-07c932f9f41e/volumes" Nov 03 20:19:39.246223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:39.246146 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:19:43.852243 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:19:43.852203 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:19:44.129064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.128982 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk"] Nov 03 20:19:44.129587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.129572 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a8d492ac-c387-473a-9c5b-17b652e7455d" containerName="revoke-keys" Nov 03 20:19:44.129648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.129589 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d492ac-c387-473a-9c5b-17b652e7455d" containerName="revoke-keys" Nov 03 20:19:44.129707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.129699 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d492ac-c387-473a-9c5b-17b652e7455d" containerName="revoke-keys" Nov 03 20:19:44.134630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.134609 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.140024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.139245 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:19:44.140727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.140261 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:19:44.143354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.143330 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk"] Nov 03 20:19:44.252612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.252584 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.252759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.252618 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdxj\" (UniqueName: \"kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.252829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.252780 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.354011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.353974 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.354146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.354022 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdxj\" (UniqueName: \"kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.354146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.354122 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.354780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.354761 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.356494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.356476 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.366436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.366413 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdxj\" (UniqueName: \"kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj\") pod \"maas-api-revoke-sub-4eb77820-l5rrk\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.469639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.460760 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:44.789015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:44.788988 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk"] Nov 03 20:19:45.496889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:45.496849 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" event={"ID":"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3","Type":"ContainerStarted","Data":"19fc447ea48a16ad06e3aac667aaacd0aedd4b4bdfac28cb49772c80280abfb1"} Nov 03 20:19:45.497373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:45.496919 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" event={"ID":"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3","Type":"ContainerStarted","Data":"2d851ff5d438f9a43640d30e9620b05a058302cd36d897174f7d3a21dbfdea98"} Nov 03 20:19:45.518406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:45.518358 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" podStartSLOduration=1.518344296 podStartE2EDuration="1.518344296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:45.515295834 +0000 UTC m=+1095.045641507" watchObservedRunningTime="2026-11-03 20:19:45.518344296 +0000 UTC m=+1095.048689968" Nov 03 20:19:46.509094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:46.509056 5815 generic.go:356] "Generic (PLEG): container finished" podID="24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" containerID="19fc447ea48a16ad06e3aac667aaacd0aedd4b4bdfac28cb49772c80280abfb1" exitCode=0 Nov 03 20:19:46.509579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:46.509146 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" event={"ID":"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3","Type":"ContainerDied","Data":"19fc447ea48a16ad06e3aac667aaacd0aedd4b4bdfac28cb49772c80280abfb1"} Nov 03 20:19:47.683358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.683328 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm"] Nov 03 20:19:47.721657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.721622 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.726144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.726113 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm"] Nov 03 20:19:47.726360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.726330 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1127f5-openshift-default-dockercfg-tnds9\"" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.850858 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852013 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852065 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852126 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852212 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852322 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852348 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vf8\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852405 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.854641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.852455 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:47.863535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:47.856555 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.016546 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\") pod \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.016600 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj\" (UniqueName: \"kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj\") pod \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.016927 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca\") pod \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017165 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017275 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017304 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vf8\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017364 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017412 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017483 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017587 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017622 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.017678 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.029547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.018271 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.038955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.038926 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.040298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.038990 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.041069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.041047 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca" pod "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" (UID: "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:48.041871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.041851 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.042593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.042571 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.144471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.144356 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.144982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.144958 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.145650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.145629 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.148896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.148629 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca" pod "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" (UID: "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:48.148896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.148864 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj" pod "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" (UID: "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3"). InnerVolumeSpecName "kube-api-access-9xdxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:48.154536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.149303 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vf8\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8\") pod \"e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.163536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.161092 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\") pod \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\" (UID: \"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3\") " Nov 03 20:19:48.170728 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:19:48.170699 5815 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 03 20:19:48.170935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.170913 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca" pod "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" (UID: "24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:19:48.181063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.181016 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:48.184296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.184257 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:48.184522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.184491 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xdxj\" (UniqueName: \"kubernetes.io/projected/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3-kube-api-access-9xdxj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:48.410668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.410588 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:48.539652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.539613 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" event={"ID":"24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3","Type":"ContainerDied","Data":"2d851ff5d438f9a43640d30e9620b05a058302cd36d897174f7d3a21dbfdea98"} Nov 03 20:19:48.539789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.539675 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d851ff5d438f9a43640d30e9620b05a058302cd36d897174f7d3a21dbfdea98" Nov 03 20:19:48.539789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.539750 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk" Nov 03 20:19:48.781069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.781034 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm"] Nov 03 20:19:48.784303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.784259 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:48.784412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.784330 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:48.784412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:48.784363 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:19:49.550080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:49.550052 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" event={"ID":"adbff6ba-33f0-4249-8fa1-6a86d9a0e194","Type":"ContainerStarted","Data":"4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e"} Nov 03 20:19:49.550260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:49.550101 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" event={"ID":"adbff6ba-33f0-4249-8fa1-6a86d9a0e194","Type":"ContainerStarted","Data":"ddc2ddedfb1e429445c10ba49880f4478981c1c6c039b9ffda4b9fbea81663e9"} Nov 03 20:19:49.588304 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:49.588252 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podStartSLOduration=2.588235616 podStartE2EDuration="2.588235616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:49.579775966 +0000 UTC m=+1099.110121639" watchObservedRunningTime="2026-11-03 20:19:49.588235616 +0000 UTC m=+1099.118581290" Nov 03 20:19:50.444541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:50.422865 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:50.652405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:50.652338 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:50.653015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:50.652997 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:50.654326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:50.654303 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:19:57.224552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.219020 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6"] Nov 03 20:19:57.224552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.219645 5815 cpu_manager.go:414] "containerMap: removing container" podUID="24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" containerName="revoke-keys" Nov 03 20:19:57.224552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.219659 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" containerName="revoke-keys" Nov 03 20:19:57.224552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.219768 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" containerName="revoke-keys" Nov 03 20:19:57.228817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.228767 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.243767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.243738 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b7b5e\"" Nov 03 20:19:57.243920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.243867 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b7b5e-dockercfg-gpxjn\"" Nov 03 20:19:57.244982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.244921 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6"] Nov 03 20:19:57.252974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.252946 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m"] Nov 03 20:19:57.259079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.259058 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.261606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.261590 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m"] Nov 03 20:19:57.418531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.409149 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.418531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.409285 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.418531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.409382 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhzc\" (UniqueName: \"kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.418531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.409426 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qv2v\" (UniqueName: \"kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v\") pod \"payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.517702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.514357 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.517702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.514482 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhzc\" (UniqueName: \"kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.517702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.514546 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qv2v\" (UniqueName: \"kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v\") pod \"payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.517702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.514667 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.517702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.515578 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.517702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.516374 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.575528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.575411 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhzc\" (UniqueName: \"kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.579603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.579575 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qv2v\" (UniqueName: \"kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v\") pod \"payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:57.579829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.579641 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:19:57.932246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:57.930165 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:19:58.099718 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:19:58.099679 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de6ff33_6146_4f7e_bdad_08d911e6b187.slice/crio-5405836cfa52d74dd61a3b3a3efe70ada6fc1149b594354c1c1591474987d3cf WatchSource:0}: Error finding container 5405836cfa52d74dd61a3b3a3efe70ada6fc1149b594354c1c1591474987d3cf: Status 404 returned error can't find the container with id 5405836cfa52d74dd61a3b3a3efe70ada6fc1149b594354c1c1591474987d3cf Nov 03 20:19:58.101703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:58.101340 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m"] Nov 03 20:19:58.331334 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:19:58.331301 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f27ca6_8c88_45ae_a221_d51068dcd6ad.slice/crio-a8a4457b135d4a2e3389e92761220e5ab4f1d0a22d390355a149ea292adcc6cb WatchSource:0}: Error finding container a8a4457b135d4a2e3389e92761220e5ab4f1d0a22d390355a149ea292adcc6cb: Status 404 returned error can't find the container with id a8a4457b135d4a2e3389e92761220e5ab4f1d0a22d390355a149ea292adcc6cb Nov 03 20:19:58.331955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:58.331910 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6"] Nov 03 20:19:59.032497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:59.032468 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" event={"ID":"09f27ca6-8c88-45ae-a221-d51068dcd6ad","Type":"ContainerStarted","Data":"a8a4457b135d4a2e3389e92761220e5ab4f1d0a22d390355a149ea292adcc6cb"} Nov 03 20:19:59.034845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:19:59.034785 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" event={"ID":"3de6ff33-6146-4f7e-bdad-08d911e6b187","Type":"ContainerStarted","Data":"5405836cfa52d74dd61a3b3a3efe70ada6fc1149b594354c1c1591474987d3cf"} Nov 03 20:20:02.102854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:02.102816 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" event={"ID":"09f27ca6-8c88-45ae-a221-d51068dcd6ad","Type":"ContainerStarted","Data":"6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66"} Nov 03 20:20:02.103239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:02.102965 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:20:02.129155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:02.129099 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" podStartSLOduration=1.512808305 podStartE2EDuration="5.129080737s" totalImagesPullingTime="3.616272432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:02.121253947 +0000 UTC m=+1111.651599624" watchObservedRunningTime="2026-11-03 20:20:02.129080737 +0000 UTC m=+1111.659426409" Nov 03 20:20:02.659390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:02.659356 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd"] Nov 03 20:20:02.659614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:02.659403 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b65143e-77qcd"] Nov 03 20:20:03.114085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:03.114040 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" event={"ID":"3de6ff33-6146-4f7e-bdad-08d911e6b187","Type":"ContainerStarted","Data":"93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867"} Nov 03 20:20:03.114650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:03.114168 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:20:03.137278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:03.136315 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" podStartSLOduration=2.236871561 podStartE2EDuration="6.136297673s" totalImagesPullingTime="3.899426112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:03.131492026 +0000 UTC m=+1112.661837691" watchObservedRunningTime="2026-11-03 20:20:03.136297673 +0000 UTC m=+1112.666643345" Nov 03 20:20:04.589233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:04.589202 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d492ac-c387-473a-9c5b-17b652e7455d" path="/var/lib/kubelet/pods/a8d492ac-c387-473a-9c5b-17b652e7455d/volumes" Nov 03 20:20:10.021734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.021701 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h"] Nov 03 20:20:10.027354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.027334 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59"] Nov 03 20:20:10.027480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.027446 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.031209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.031189 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9b65143e\"" Nov 03 20:20:10.031630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.031610 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.040191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.040075 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h"] Nov 03 20:20:10.040562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.040543 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9b65143e-dockercfg-vxqd8\"" Nov 03 20:20:10.046072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.046048 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59"] Nov 03 20:20:10.115970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.115893 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.116158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.115987 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvbz\" (UniqueName: \"kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-vns59\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.116158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.116100 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-vns59\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.116284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.116183 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jhtv\" (UniqueName: \"kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.221386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.221347 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jhtv\" (UniqueName: \"kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.237877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.237843 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.238125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.238108 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvbz\" (UniqueName: \"kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-vns59\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.238308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.238293 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-vns59\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.239297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.239272 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-vns59\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.254011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.244270 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.266838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.266808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvbz\" (UniqueName: \"kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-vns59\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.287344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.287241 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jhtv\" (UniqueName: \"kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.355608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.355309 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:10.373072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.369969 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:10.845723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.840153 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h"] Nov 03 20:20:10.936198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:10.936160 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59"] Nov 03 20:20:11.030346 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:20:11.028981 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea1f570_0d09_4add_b1a0_6bf2bc1c053c.slice/crio-f7fdb0dce780eace12682057aa50c08e506f45cdb34bf5b4abeaff266f5010f3 WatchSource:0}: Error finding container f7fdb0dce780eace12682057aa50c08e506f45cdb34bf5b4abeaff266f5010f3: Status 404 returned error can't find the container with id f7fdb0dce780eace12682057aa50c08e506f45cdb34bf5b4abeaff266f5010f3 Nov 03 20:20:11.260558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.260491 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" event={"ID":"fea1f570-0d09-4add-b1a0-6bf2bc1c053c","Type":"ContainerStarted","Data":"32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584"} Nov 03 20:20:11.260746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.260612 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:11.260746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.260631 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" event={"ID":"fea1f570-0d09-4add-b1a0-6bf2bc1c053c","Type":"ContainerStarted","Data":"f7fdb0dce780eace12682057aa50c08e506f45cdb34bf5b4abeaff266f5010f3"} Nov 03 20:20:11.262646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.262619 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" event={"ID":"e5a46758-59f5-434f-81dd-c01fda609e7a","Type":"ContainerStarted","Data":"9177cebb14ca653011ab21d27fb61bca3b5b92704d8c263fe21b88d48ab71dad"} Nov 03 20:20:11.262779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.262660 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" event={"ID":"e5a46758-59f5-434f-81dd-c01fda609e7a","Type":"ContainerStarted","Data":"6b8ccce8b016148d4e355d0e7f3e2e7d89426de199b47be0d040d67135ca116b"} Nov 03 20:20:11.262779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.262728 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:11.280643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.280576 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" podStartSLOduration=1.2805609439999999 podStartE2EDuration="1.280560944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:11.278669683 +0000 UTC m=+1120.809015355" watchObservedRunningTime="2026-11-03 20:20:11.280560944 +0000 UTC m=+1120.810906628" Nov 03 20:20:11.303412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:11.303367 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" podStartSLOduration=1.303354412 podStartE2EDuration="1.303354412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:11.297442693 +0000 UTC m=+1120.827788364" watchObservedRunningTime="2026-11-03 20:20:11.303354412 +0000 UTC m=+1120.833700105" Nov 03 20:20:13.117880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:13.117855 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:20:13.852025 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:13.851988 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:20:14.128612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:14.128542 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:20:16.357038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:16.357007 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:20:17.876236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.876200 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9"] Nov 03 20:20:17.880727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.880701 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:17.888672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.888647 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq"] Nov 03 20:20:17.889586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.889549 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b9849\"" Nov 03 20:20:17.889865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.889842 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b9849-dockercfg-ppgrz\"" Nov 03 20:20:17.895703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.895682 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:17.896451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.896431 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9"] Nov 03 20:20:17.904556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.904535 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq"] Nov 03 20:20:17.973533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:17.958341 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbj8v\" (UniqueName: \"kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v\") pod \"payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.021755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.015654 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.021755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.015887 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.021755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.015998 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wm5x\" (UniqueName: \"kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.124533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.123587 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wm5x\" (UniqueName: \"kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.124533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.123711 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbj8v\" (UniqueName: \"kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v\") pod \"payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.124533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.123778 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.124533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.123924 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.124891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.124838 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.167598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.126120 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.234660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.215284 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbj8v\" (UniqueName: \"kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v\") pod \"payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.234660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.224785 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:18.239528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.236148 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wm5x\" (UniqueName: \"kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.520324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.520284 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:18.744602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:18.744313 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq"] Nov 03 20:20:19.451798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:19.451771 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9"] Nov 03 20:20:19.468841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:19.468809 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" event={"ID":"5c797e9f-208a-4378-97c8-78a93156f86a","Type":"ContainerStarted","Data":"49f7bc0f47e45832f10bcf8bcf2e61a8f676a3a3156a71394ec4e0e4082e3328"} Nov 03 20:20:20.521219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:20.521181 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" event={"ID":"a07f22ff-a809-4060-a9d8-74944518d18e","Type":"ContainerStarted","Data":"c892bb43e05c97559ed585af3ab195a868a8b308fb3ac5424cf5c75838c45aae"} Nov 03 20:20:21.086656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:21.086614 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6"] Nov 03 20:20:21.117776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:21.116963 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m"] Nov 03 20:20:22.312562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:22.312136 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:20:22.313166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:22.312854 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:20:23.633082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:23.633053 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" event={"ID":"5c797e9f-208a-4378-97c8-78a93156f86a","Type":"ContainerStarted","Data":"baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356"} Nov 03 20:20:23.633881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:23.633864 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:23.657858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:23.657800 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" podStartSLOduration=2.474738465 podStartE2EDuration="6.657783169s" totalImagesPullingTime="4.183044704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:23.652655022 +0000 UTC m=+1133.183000695" watchObservedRunningTime="2026-11-03 20:20:23.657783169 +0000 UTC m=+1133.188128841" Nov 03 20:20:24.643102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:24.643069 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" event={"ID":"a07f22ff-a809-4060-a9d8-74944518d18e","Type":"ContainerStarted","Data":"cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b"} Nov 03 20:20:24.643570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:24.643296 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:24.669244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:24.669192 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" podStartSLOduration=3.505735381 podStartE2EDuration="7.669174566s" totalImagesPullingTime="4.163439185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:24.661377304 +0000 UTC m=+1134.191722980" watchObservedRunningTime="2026-11-03 20:20:24.669174566 +0000 UTC m=+1134.199520247" Nov 03 20:20:26.088571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.088039 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" podUID="09f27ca6-8c88-45ae-a221-d51068dcd6ad" containerName="payload-processing" containerID="cri-o://6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66" gracePeriod=25 Nov 03 20:20:26.118242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.118208 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" podUID="3de6ff33-6146-4f7e-bdad-08d911e6b187" containerName="payload-pre-processing" containerID="cri-o://93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867" gracePeriod=25 Nov 03 20:20:26.533858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.533491 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:20:26.565007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.564971 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:20:26.613087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.612943 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v\" (UniqueName: \"kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v\") pod \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " Nov 03 20:20:26.613237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.613184 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume\") pod \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\" (UID: \"09f27ca6-8c88-45ae-a221-d51068dcd6ad\") " Nov 03 20:20:26.614100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.613915 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume" pod "09f27ca6-8c88-45ae-a221-d51068dcd6ad" (UID: "09f27ca6-8c88-45ae-a221-d51068dcd6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:26.618488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.616776 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v" pod "09f27ca6-8c88-45ae-a221-d51068dcd6ad" (UID: "09f27ca6-8c88-45ae-a221-d51068dcd6ad"). InnerVolumeSpecName "kube-api-access-8qv2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:26.664607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.664574 5815 generic.go:356] "Generic (PLEG): container finished" podID="3de6ff33-6146-4f7e-bdad-08d911e6b187" containerID="93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867" exitCode=0 Nov 03 20:20:26.664767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.664625 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" Nov 03 20:20:26.664767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.664666 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" event={"ID":"3de6ff33-6146-4f7e-bdad-08d911e6b187","Type":"ContainerDied","Data":"93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867"} Nov 03 20:20:26.664767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.664734 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m" event={"ID":"3de6ff33-6146-4f7e-bdad-08d911e6b187","Type":"ContainerDied","Data":"5405836cfa52d74dd61a3b3a3efe70ada6fc1149b594354c1c1591474987d3cf"} Nov 03 20:20:26.664767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.664766 5815 scope.go:122] "RemoveContainer" containerID="93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867" Nov 03 20:20:26.667038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.667015 5815 generic.go:356] "Generic (PLEG): container finished" podID="09f27ca6-8c88-45ae-a221-d51068dcd6ad" containerID="6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66" exitCode=0 Nov 03 20:20:26.667151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.667105 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" Nov 03 20:20:26.669531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.667146 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" event={"ID":"09f27ca6-8c88-45ae-a221-d51068dcd6ad","Type":"ContainerDied","Data":"6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66"} Nov 03 20:20:26.669531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.667848 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6" event={"ID":"09f27ca6-8c88-45ae-a221-d51068dcd6ad","Type":"ContainerDied","Data":"a8a4457b135d4a2e3389e92761220e5ab4f1d0a22d390355a149ea292adcc6cb"} Nov 03 20:20:26.681985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.681959 5815 scope.go:122] "RemoveContainer" containerID="93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867" Nov 03 20:20:26.682545 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:26.682500 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867\": container with ID starting with 93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867 not found: ID does not exist" containerID="93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867" Nov 03 20:20:26.683396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.683349 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867"} err="failed to get container status \"93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867\": rpc error: code = NotFound desc = could not find container \"93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867\": container with ID starting with 93651a0fcfc529ab9040889300d75a2d7f038dc8f11f6d7e9152ea39a0ea0867 not found: ID does not exist" Nov 03 20:20:26.683567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.683553 5815 scope.go:122] "RemoveContainer" containerID="6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66" Nov 03 20:20:26.700596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.700493 5815 scope.go:122] "RemoveContainer" containerID="6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66" Nov 03 20:20:26.700989 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:26.700961 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66\": container with ID starting with 6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66 not found: ID does not exist" containerID="6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66" Nov 03 20:20:26.701078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.701001 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66"} err="failed to get container status \"6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66\": rpc error: code = NotFound desc = could not find container \"6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66\": container with ID starting with 6a894f90f91f8d007b3317080aa8fe2d55a7b101c712092322878869ffaf8e66 not found: ID does not exist" Nov 03 20:20:26.706080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.706057 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6"] Nov 03 20:20:26.710621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.710601 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-6787dbf956-8hsb6"] Nov 03 20:20:26.716765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.714703 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc\" (UniqueName: \"kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc\") pod \"3de6ff33-6146-4f7e-bdad-08d911e6b187\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " Nov 03 20:20:26.716765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.714826 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume\") pod \"3de6ff33-6146-4f7e-bdad-08d911e6b187\" (UID: \"3de6ff33-6146-4f7e-bdad-08d911e6b187\") " Nov 03 20:20:26.716765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.715430 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f27ca6-8c88-45ae-a221-d51068dcd6ad-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:26.716765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.715452 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qv2v\" (UniqueName: \"kubernetes.io/projected/09f27ca6-8c88-45ae-a221-d51068dcd6ad-kube-api-access-8qv2v\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:26.716765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.715863 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume" pod "3de6ff33-6146-4f7e-bdad-08d911e6b187" (UID: "3de6ff33-6146-4f7e-bdad-08d911e6b187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:26.717212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.717192 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc" pod "3de6ff33-6146-4f7e-bdad-08d911e6b187" (UID: "3de6ff33-6146-4f7e-bdad-08d911e6b187"). InnerVolumeSpecName "kube-api-access-lxhzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:26.816425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.816390 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de6ff33-6146-4f7e-bdad-08d911e6b187-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:26.816425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.816418 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxhzc\" (UniqueName: \"kubernetes.io/projected/3de6ff33-6146-4f7e-bdad-08d911e6b187-kube-api-access-lxhzc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:26.996922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:26.996458 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m"] Nov 03 20:20:27.004053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.004023 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-778bd8ddc9-qzx9m"] Nov 03 20:20:27.362081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362010 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7570-1a30a2"] Nov 03 20:20:27.362656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362641 5815 cpu_manager.go:414] "containerMap: removing container" podUID="09f27ca6-8c88-45ae-a221-d51068dcd6ad" containerName="payload-processing" Nov 03 20:20:27.362714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362658 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f27ca6-8c88-45ae-a221-d51068dcd6ad" containerName="payload-processing" Nov 03 20:20:27.362714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362685 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3de6ff33-6146-4f7e-bdad-08d911e6b187" containerName="payload-pre-processing" Nov 03 20:20:27.362714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362690 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de6ff33-6146-4f7e-bdad-08d911e6b187" containerName="payload-pre-processing" Nov 03 20:20:27.362815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362800 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de6ff33-6146-4f7e-bdad-08d911e6b187" containerName="payload-pre-processing" Nov 03 20:20:27.362845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.362814 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f27ca6-8c88-45ae-a221-d51068dcd6ad" containerName="payload-processing" Nov 03 20:20:27.368436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.368220 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:27.373092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.373072 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6p5pq\"" Nov 03 20:20:27.378934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.378907 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7570-1a30a2"] Nov 03 20:20:27.526549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.526488 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgvs\" (UniqueName: \"kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs\") pod \"test-curl-7570-1a30a2\" (UID: \"68e54767-4d56-49da-8a4a-82a2fd708cbe\") " pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:27.628166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.628069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgvs\" (UniqueName: \"kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs\") pod \"test-curl-7570-1a30a2\" (UID: \"68e54767-4d56-49da-8a4a-82a2fd708cbe\") " pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:27.640359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.640337 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgvs\" (UniqueName: \"kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs\") pod \"test-curl-7570-1a30a2\" (UID: \"68e54767-4d56-49da-8a4a-82a2fd708cbe\") " pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:27.692971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:27.692919 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:28.208937 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:20:28.208115 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68e54767_4d56_49da_8a4a_82a2fd708cbe.slice/crio-1fa6e8f6a1f3fe75d4d4dee306a58dec926d80a00e32cdcae249b9c9bb72d602 WatchSource:0}: Error finding container 1fa6e8f6a1f3fe75d4d4dee306a58dec926d80a00e32cdcae249b9c9bb72d602: Status 404 returned error can't find the container with id 1fa6e8f6a1f3fe75d4d4dee306a58dec926d80a00e32cdcae249b9c9bb72d602 Nov 03 20:20:28.208937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:28.208423 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7570-1a30a2"] Nov 03 20:20:28.587123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:28.587088 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f27ca6-8c88-45ae-a221-d51068dcd6ad" path="/var/lib/kubelet/pods/09f27ca6-8c88-45ae-a221-d51068dcd6ad/volumes" Nov 03 20:20:28.587773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:28.587751 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de6ff33-6146-4f7e-bdad-08d911e6b187" path="/var/lib/kubelet/pods/3de6ff33-6146-4f7e-bdad-08d911e6b187/volumes" Nov 03 20:20:28.698501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:28.698444 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7570-1a30a2" event={"ID":"68e54767-4d56-49da-8a4a-82a2fd708cbe","Type":"ContainerStarted","Data":"f47edfd56e61712a8a79ba27655d684f38f4f99a9f8641e0e15b36fd69ca9f2d"} Nov 03 20:20:28.698501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:28.698496 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7570-1a30a2" event={"ID":"68e54767-4d56-49da-8a4a-82a2fd708cbe","Type":"ContainerStarted","Data":"1fa6e8f6a1f3fe75d4d4dee306a58dec926d80a00e32cdcae249b9c9bb72d602"} Nov 03 20:20:28.716485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:28.716437 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7570-1a30a2" podStartSLOduration=1.71642516 podStartE2EDuration="1.71642516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:28.714487445 +0000 UTC m=+1138.244833118" watchObservedRunningTime="2026-11-03 20:20:28.71642516 +0000 UTC m=+1138.246770832" Nov 03 20:20:29.508779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.508737 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7570-1a30a2"] Nov 03 20:20:29.513665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.513637 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7570-1a30a2"] Nov 03 20:20:29.710338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.710310 5815 generic.go:356] "Generic (PLEG): container finished" podID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerID="4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e" exitCode=0 Nov 03 20:20:29.710782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.710386 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" event={"ID":"adbff6ba-33f0-4249-8fa1-6a86d9a0e194","Type":"ContainerDied","Data":"4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e"} Nov 03 20:20:29.710880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.710863 5815 scope.go:122] "RemoveContainer" containerID="4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e" Nov 03 20:20:29.712498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.712466 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:29.712640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.712570 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:29.712640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:29.712615 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:30.722081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:30.722048 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" event={"ID":"adbff6ba-33f0-4249-8fa1-6a86d9a0e194","Type":"ContainerStarted","Data":"7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada"} Nov 03 20:20:30.722600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:30.722131 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7570-1a30a2" podUID="68e54767-4d56-49da-8a4a-82a2fd708cbe" containerName="test-curl-7570-1a30a2" containerID="cri-o://f47edfd56e61712a8a79ba27655d684f38f4f99a9f8641e0e15b36fd69ca9f2d" gracePeriod=2 Nov 03 20:20:31.411397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:31.411357 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:20:31.412434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:31.412412 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:20:31.412586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:31.412468 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 20:20:32.085078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.085047 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2"] Nov 03 20:20:32.126669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.125900 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a50d7ad-z9zc2"] Nov 03 20:20:32.318701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.311531 5815 scope.go:122] "RemoveContainer" containerID="0438d9a58f947b01063972b691cbc1118a010579874c8f3b639a0d047eedd678" Nov 03 20:20:32.415836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.405709 5815 scope.go:122] "RemoveContainer" containerID="ca4dec2cbe718b4111332de76f15c22510a291b7ab6f0ef43f9df799177039d5" Nov 03 20:20:32.415924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.415845 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:20:32.415924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.415908 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 20:20:32.463155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.463128 5815 scope.go:122] "RemoveContainer" containerID="72441ff27637ca49e46f8033b8c4f8c95396275f695471701c042d0aeef64c1b" Nov 03 20:20:32.508767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.507426 5815 scope.go:122] "RemoveContainer" containerID="9d2bfdfa4e8dfc8ee54dd6e199055f5413267aa58f80e837c7c6f16eacc05dbe" Nov 03 20:20:32.525986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.525962 5815 scope.go:122] "RemoveContainer" containerID="cdef2c9835372a8f0c4066a10601634d3ada52334abb59176aa28c5f4f7580d1" Nov 03 20:20:32.539025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.538998 5815 scope.go:122] "RemoveContainer" containerID="58cd479b94ecb3da8fc80c9321217e0a2f6bcdc5d4ec72a8ff2f65da485238e4" Nov 03 20:20:32.554097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.554075 5815 scope.go:122] "RemoveContainer" containerID="f9beb736435fb4d6074b28cd9f27584b3dba496cc9c6fd23261eb76a4779d7bf" Nov 03 20:20:32.564176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.564157 5815 scope.go:122] "RemoveContainer" containerID="8a0e7229741da64e2bec3be0e142d95ac087555cd30f578d30b6ea72506df97b" Nov 03 20:20:32.616594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.612900 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1741ef8-0dd2-48a2-b41b-e2576488eeb3" path="/var/lib/kubelet/pods/e1741ef8-0dd2-48a2-b41b-e2576488eeb3/volumes" Nov 03 20:20:32.852887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:32.852839 5815 generic.go:356] "Generic (PLEG): container finished" podID="68e54767-4d56-49da-8a4a-82a2fd708cbe" containerID="f47edfd56e61712a8a79ba27655d684f38f4f99a9f8641e0e15b36fd69ca9f2d" exitCode=137 Nov 03 20:20:33.090532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.086579 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:33.243535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.242462 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs\" (UniqueName: \"kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs\") pod \"68e54767-4d56-49da-8a4a-82a2fd708cbe\" (UID: \"68e54767-4d56-49da-8a4a-82a2fd708cbe\") " Nov 03 20:20:33.247267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.246828 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs" pod "68e54767-4d56-49da-8a4a-82a2fd708cbe" (UID: "68e54767-4d56-49da-8a4a-82a2fd708cbe"). InnerVolumeSpecName "kube-api-access-mtgvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:33.343875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.343843 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgvs\" (UniqueName: \"kubernetes.io/projected/68e54767-4d56-49da-8a4a-82a2fd708cbe-kube-api-access-mtgvs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:33.416078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.416042 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:20:33.416363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.416332 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 20:20:33.928746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.926074 5815 scope.go:122] "RemoveContainer" containerID="f47edfd56e61712a8a79ba27655d684f38f4f99a9f8641e0e15b36fd69ca9f2d" Nov 03 20:20:33.928746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:33.926208 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:33.938733 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:33.938707 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7570-1a30a2\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="68e54767-4d56-49da-8a4a-82a2fd708cbe" pod="opendatahub/test-curl-7570-1a30a2" Nov 03 20:20:34.411474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:34.411444 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:20:34.411889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:34.411499 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 20:20:34.606026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:34.605992 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e54767-4d56-49da-8a4a-82a2fd708cbe" path="/var/lib/kubelet/pods/68e54767-4d56-49da-8a4a-82a2fd708cbe/volumes" Nov 03 20:20:34.996453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:34.996416 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c"] Nov 03 20:20:35.411319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:35.411238 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:20:35.411471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:35.411306 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 20:20:35.653035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:35.652995 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:20:35.655047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:35.655021 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:20:35.655206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:35.655138 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:20:36.411129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:36.411090 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:20:36.411337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:36.411144 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 20:20:37.415793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:37.415759 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:20:37.417093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:37.417069 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:20:43.907487 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:43.907443 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:20:45.566610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:45.566577 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:20:46.050491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:46.050458 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg"] Nov 03 20:20:46.054859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:46.054832 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d33a5ba4-dxdgg"] Nov 03 20:20:46.599327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:46.598875 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997f91a2-55e4-4c99-b40a-fabd64ee5d44" path="/var/lib/kubelet/pods/997f91a2-55e4-4c99-b40a-fabd64ee5d44/volumes" Nov 03 20:20:47.178329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:47.178232 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk"] Nov 03 20:20:47.178608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:47.178593 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e6cc3e-pfsbk"] Nov 03 20:20:48.255456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.255416 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8"] Nov 03 20:20:48.256358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.256335 5815 cpu_manager.go:414] "containerMap: removing container" podUID="68e54767-4d56-49da-8a4a-82a2fd708cbe" containerName="test-curl-7570-1a30a2" Nov 03 20:20:48.256457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.256361 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e54767-4d56-49da-8a4a-82a2fd708cbe" containerName="test-curl-7570-1a30a2" Nov 03 20:20:48.256634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.256619 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e54767-4d56-49da-8a4a-82a2fd708cbe" containerName="test-curl-7570-1a30a2" Nov 03 20:20:48.261416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.261391 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.290935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.286775 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d6789f-openshift-default-dockercfg-hglfr\"" Nov 03 20:20:48.290935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.289379 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8"] Nov 03 20:20:48.346230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.346180 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvj4\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.346558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.346535 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.346776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.346760 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.346932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.346918 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.347072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.347055 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.347284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.347254 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.347350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.347311 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.347350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.347340 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.347471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.347436 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.465550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.465488 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.482426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.482394 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvj4\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.482725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.482705 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.482957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.482936 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.483117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.483098 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.483296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.483278 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.483442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.483422 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.483646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.483626 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.483819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.483798 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.484404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.467019 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.499632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.484863 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.499832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.485346 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.499932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.486182 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.500022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.486919 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.511259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.511140 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.521208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.521179 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvj4\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.535397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.528271 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.535685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.532103 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy\") pod \"e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.600823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.600785 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:48.636077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:48.635429 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc17f05-6452-45c0-a405-19f71acfc3d6" path="/var/lib/kubelet/pods/9fc17f05-6452-45c0-a405-19f71acfc3d6/volumes" Nov 03 20:20:49.086737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.086706 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8"] Nov 03 20:20:49.087621 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:20:49.087590 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb070e8_cb4e_467b_b35d_5c8b0a77e475.slice/crio-c1daa7f1d8e496aa805fe3639fabb893bcb03aee8606d0c1bf7652181ad7bcae WatchSource:0}: Error finding container c1daa7f1d8e496aa805fe3639fabb893bcb03aee8606d0c1bf7652181ad7bcae: Status 404 returned error can't find the container with id c1daa7f1d8e496aa805fe3639fabb893bcb03aee8606d0c1bf7652181ad7bcae Nov 03 20:20:49.091015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.090979 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:49.091119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.091075 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:49.091119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.091107 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:49.219563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.217030 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" event={"ID":"bfb070e8-cb4e-467b-b35d-5c8b0a77e475","Type":"ContainerStarted","Data":"09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162"} Nov 03 20:20:49.219563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.217090 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" event={"ID":"bfb070e8-cb4e-467b-b35d-5c8b0a77e475","Type":"ContainerStarted","Data":"c1daa7f1d8e496aa805fe3639fabb893bcb03aee8606d0c1bf7652181ad7bcae"} Nov 03 20:20:49.244321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.244261 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podStartSLOduration=1.244240074 podStartE2EDuration="1.244240074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:49.239676512 +0000 UTC m=+1158.770022185" watchObservedRunningTime="2026-11-03 20:20:49.244240074 +0000 UTC m=+1158.774585747" Nov 03 20:20:49.610531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.606379 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:49.763116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.762954 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm"] Nov 03 20:20:49.763295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:49.763227 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" containerID="cri-o://7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada" gracePeriod=30 Nov 03 20:20:50.037305 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:50.037254 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerName="main" containerID="cri-o://b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c" gracePeriod=45 Nov 03 20:20:50.216745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:50.212902 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": EOF" Nov 03 20:20:50.610685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:50.610655 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:51.021040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.021015 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:20:51.060248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060219 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir\") pod \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " Nov 03 20:20:51.060415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060285 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home\") pod \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " Nov 03 20:20:51.060488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060444 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct\" (UniqueName: \"kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct\") pod \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " Nov 03 20:20:51.060488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060482 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs\") pod \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " Nov 03 20:20:51.060629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060481 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir" pod "6ea24e8f-5b74-4ddd-b7fc-810009660e94" (UID: "6ea24e8f-5b74-4ddd-b7fc-810009660e94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:51.060629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060623 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm\") pod \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " Nov 03 20:20:51.060816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060622 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home" pod "6ea24e8f-5b74-4ddd-b7fc-810009660e94" (UID: "6ea24e8f-5b74-4ddd-b7fc-810009660e94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:51.060816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.060659 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache\") pod \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\" (UID: \"6ea24e8f-5b74-4ddd-b7fc-810009660e94\") " Nov 03 20:20:51.061341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.061322 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:51.061441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.061345 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:51.061621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.061603 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache" pod "6ea24e8f-5b74-4ddd-b7fc-810009660e94" (UID: "6ea24e8f-5b74-4ddd-b7fc-810009660e94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:51.064307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.064268 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs" pod "6ea24e8f-5b74-4ddd-b7fc-810009660e94" (UID: "6ea24e8f-5b74-4ddd-b7fc-810009660e94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:51.064307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.064260 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm" pod "6ea24e8f-5b74-4ddd-b7fc-810009660e94" (UID: "6ea24e8f-5b74-4ddd-b7fc-810009660e94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:51.079806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.079781 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct" pod "6ea24e8f-5b74-4ddd-b7fc-810009660e94" (UID: "6ea24e8f-5b74-4ddd-b7fc-810009660e94"). InnerVolumeSpecName "kube-api-access-psgct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:51.163171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.163099 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psgct\" (UniqueName: \"kubernetes.io/projected/6ea24e8f-5b74-4ddd-b7fc-810009660e94-kube-api-access-psgct\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:51.163171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.163124 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea24e8f-5b74-4ddd-b7fc-810009660e94-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:51.163171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.163134 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:51.163171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.163142 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea24e8f-5b74-4ddd-b7fc-810009660e94-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:51.268162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.268126 5815 generic.go:356] "Generic (PLEG): container finished" podID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerID="b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c" exitCode=0 Nov 03 20:20:51.268353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.268178 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" Nov 03 20:20:51.268353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.268216 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" event={"ID":"6ea24e8f-5b74-4ddd-b7fc-810009660e94","Type":"ContainerDied","Data":"b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c"} Nov 03 20:20:51.268353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.268264 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" event={"ID":"6ea24e8f-5b74-4ddd-b7fc-810009660e94","Type":"ContainerDied","Data":"5c82d8b2fc600a8d32ba36377e3a9c44dab68bc00af74858bb2cebaa8fdae8bf"} Nov 03 20:20:51.268353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.268286 5815 scope.go:122] "RemoveContainer" containerID="b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c" Nov 03 20:20:51.269339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.269315 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:51.270459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.270436 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:20:51.281961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.281884 5815 scope.go:122] "RemoveContainer" containerID="b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c" Nov 03 20:20:51.282310 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:51.282282 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c\": container with ID starting with b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c not found: ID does not exist" containerID="b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c" Nov 03 20:20:51.282409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.282322 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c"} err="failed to get container status \"b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c\": rpc error: code = NotFound desc = could not find container \"b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c\": container with ID starting with b1f7706aea86a2c7af25352fb9c46a7497e442241a7146204db491254e85c58c not found: ID does not exist" Nov 03 20:20:51.343409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.340209 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c"] Nov 03 20:20:51.356435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.356405 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c"] Nov 03 20:20:51.458465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.458392 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8"] Nov 03 20:20:51.459415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.459394 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerName="main" Nov 03 20:20:51.459534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.459418 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerName="main" Nov 03 20:20:51.459715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.459699 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerName="main" Nov 03 20:20:51.465412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.465392 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.480026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.480004 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8"] Nov 03 20:20:51.563538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.560878 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf9b0a-openshift-default-dockercfg-25g69\"" Nov 03 20:20:51.659534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659454 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659564 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659630 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659707 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659766 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659792 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659953 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.659975 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.660073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.660042 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b64w\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.762474 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.762607 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.762684 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.762769 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.762863 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.762891 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.763109 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.763133 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.763532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.763209 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b64w\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.764158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.764028 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.767529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.767061 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.771526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.767720 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.771526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.768229 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.771526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.769191 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.775596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.775570 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.807550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.807496 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.834542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.822229 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b64w\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.834542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.833347 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token\") pod \"e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:51.956763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:51.954612 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9b65143e/e2e-lifecycle-model-9b65143e-kserve-9ff446df-n4r5c" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:20:52.088952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.088862 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:52.395735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.395574 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8"] Nov 03 20:20:52.399125 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:20:52.399082 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c3a2ff_efb9_48f9_ad69_2c935fbcb321.slice/crio-2b938c4df6eb9548644a7fa512feb7d7fc6a60633b858ceaae44c104570004ae WatchSource:0}: Error finding container 2b938c4df6eb9548644a7fa512feb7d7fc6a60633b858ceaae44c104570004ae: Status 404 returned error can't find the container with id 2b938c4df6eb9548644a7fa512feb7d7fc6a60633b858ceaae44c104570004ae Nov 03 20:20:52.401765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.401723 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:52.401863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.401816 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:52.401921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.401859 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:20:52.419241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.419212 5815 patch_prober.go:29] interesting pod/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:20:52.419347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.419269 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:20:52.608458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:52.606471 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea24e8f-5b74-4ddd-b7fc-810009660e94" path="/var/lib/kubelet/pods/6ea24e8f-5b74-4ddd-b7fc-810009660e94/volumes" Nov 03 20:20:53.354771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:53.354739 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" event={"ID":"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321","Type":"ContainerStarted","Data":"9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f"} Nov 03 20:20:53.355261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:53.354792 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" event={"ID":"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321","Type":"ContainerStarted","Data":"2b938c4df6eb9548644a7fa512feb7d7fc6a60633b858ceaae44c104570004ae"} Nov 03 20:20:53.382348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:53.382298 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podStartSLOduration=2.382284488 podStartE2EDuration="2.382284488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:53.378453717 +0000 UTC m=+1162.908799391" watchObservedRunningTime="2026-11-03 20:20:53.382284488 +0000 UTC m=+1162.912630159" Nov 03 20:20:54.090560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:54.090477 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:54.098044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:54.097459 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:54.363614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:54.363537 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:54.364407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:54.364388 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:20:55.099579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.080745 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.177970 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178100 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178149 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178179 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178347 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178459 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.178782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178565 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert\") pod \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\" (UID: \"adbff6ba-33f0-4249-8fa1-6a86d9a0e194\") " Nov 03 20:20:55.179375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.178906 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:55.179375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.179016 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:55.179375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.179283 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:55.179621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.179598 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.179692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.179633 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.179692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.179650 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.179692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.179602 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:55.181708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.181570 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "kube-api-access-24vf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:55.181708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.181680 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:20:55.184710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.183374 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:55.186024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.185988 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:55.186685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.186664 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo" pod "adbff6ba-33f0-4249-8fa1-6a86d9a0e194" (UID: "adbff6ba-33f0-4249-8fa1-6a86d9a0e194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:20:55.281449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.281417 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.281449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.281448 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vf8\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-kube-api-access-24vf8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.281675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.281464 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.281675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.281480 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.281675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.281493 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.281675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.281530 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adbff6ba-33f0-4249-8fa1-6a86d9a0e194-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:55.375190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.375109 5815 generic.go:356] "Generic (PLEG): container finished" podID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerID="7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada" exitCode=0 Nov 03 20:20:55.375190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.375158 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" Nov 03 20:20:55.375673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.375190 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" event={"ID":"adbff6ba-33f0-4249-8fa1-6a86d9a0e194","Type":"ContainerDied","Data":"7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada"} Nov 03 20:20:55.375673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.375242 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm" event={"ID":"adbff6ba-33f0-4249-8fa1-6a86d9a0e194","Type":"ContainerDied","Data":"ddc2ddedfb1e429445c10ba49880f4478981c1c6c039b9ffda4b9fbea81663e9"} Nov 03 20:20:55.375673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.375264 5815 scope.go:122] "RemoveContainer" containerID="7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada" Nov 03 20:20:55.390260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.390243 5815 scope.go:122] "RemoveContainer" containerID="4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e" Nov 03 20:20:55.409994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.409966 5815 scope.go:122] "RemoveContainer" containerID="7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada" Nov 03 20:20:55.410582 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:55.410553 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada\": container with ID starting with 7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada not found: ID does not exist" containerID="7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada" Nov 03 20:20:55.410705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.410592 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada"} err="failed to get container status \"7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada\": rpc error: code = NotFound desc = could not find container \"7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada\": container with ID starting with 7e7fde5a38b63b832244712823b40e875c0d5e959e5736946d1e6a242ee43ada not found: ID does not exist" Nov 03 20:20:55.410705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.410616 5815 scope.go:122] "RemoveContainer" containerID="4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e" Nov 03 20:20:55.410998 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:20:55.410970 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e\": container with ID starting with 4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e not found: ID does not exist" containerID="4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e" Nov 03 20:20:55.411060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.411014 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e"} err="failed to get container status \"4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e\": rpc error: code = NotFound desc = could not find container \"4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e\": container with ID starting with 4ebc72b3f66828b34caea079c65b083d4d5b43a31ee658f4fa4bfaf2b9e96c4e not found: ID does not exist" Nov 03 20:20:55.415408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.415387 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm"] Nov 03 20:20:55.429484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:55.429460 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1127f5-openshift-default-578bff5f-gdbqm"] Nov 03 20:20:56.600594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.597013 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" path="/var/lib/kubelet/pods/adbff6ba-33f0-4249-8fa1-6a86d9a0e194/volumes" Nov 03 20:20:56.751843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.746184 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv"] Nov 03 20:20:56.751843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.747131 5815 cpu_manager.go:414] "containerMap: removing container" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" Nov 03 20:20:56.751843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.747197 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" Nov 03 20:20:56.752189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.752168 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" Nov 03 20:20:56.752250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.752199 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" Nov 03 20:20:56.760580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.759138 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv"] Nov 03 20:20:56.760580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.759248 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:56.766551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.766238 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf9b0a-dockercfg-7tc5q\"" Nov 03 20:20:56.766551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.766254 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf9b0a\"" Nov 03 20:20:56.814798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.814771 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5"] Nov 03 20:20:56.815372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.815358 5815 cpu_manager.go:414] "containerMap: removing container" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" Nov 03 20:20:56.815420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.815374 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbff6ba-33f0-4249-8fa1-6a86d9a0e194" containerName="istio-proxy" Nov 03 20:20:56.819173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.819154 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:56.822998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.822969 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:56.822998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.823150 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhn9\" (UniqueName: \"kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:56.829219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.829194 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5"] Nov 03 20:20:56.926435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.926390 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:56.926676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.926565 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:56.926676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.926627 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rw9\" (UniqueName: \"kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9\") pod \"payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:56.926781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.926717 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhn9\" (UniqueName: \"kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:56.927289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:56.927261 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:57.049538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.034590 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:57.049538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.034666 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rw9\" (UniqueName: \"kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9\") pod \"payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:57.049538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.035984 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:57.133125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.133080 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rw9\" (UniqueName: \"kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9\") pod \"payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:57.160712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.160183 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhn9\" (UniqueName: \"kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:57.164606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.164580 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm"] Nov 03 20:20:57.170785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.170758 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2b5f8cc-dxjsm"] Nov 03 20:20:57.383820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.383781 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:57.443451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.443374 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:57.838801 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:20:57.838771 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4913026_0b4a_4a34_824b_e03e1a52fe40.slice/crio-49838d722b6e84ee3252e92f04f3a38c8d4fca9facfa2000753538324982cb86 WatchSource:0}: Error finding container 49838d722b6e84ee3252e92f04f3a38c8d4fca9facfa2000753538324982cb86: Status 404 returned error can't find the container with id 49838d722b6e84ee3252e92f04f3a38c8d4fca9facfa2000753538324982cb86 Nov 03 20:20:57.839856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.839837 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv"] Nov 03 20:20:57.863650 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:20:57.863617 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0d9e70_1046_4968_8f8a_b5abc530753a.slice/crio-1c62043b64154ba39304afae196d7b73513066ca9ab01f286df292033d64e0cf WatchSource:0}: Error finding container 1c62043b64154ba39304afae196d7b73513066ca9ab01f286df292033d64e0cf: Status 404 returned error can't find the container with id 1c62043b64154ba39304afae196d7b73513066ca9ab01f286df292033d64e0cf Nov 03 20:20:57.865126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:57.865105 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5"] Nov 03 20:20:58.483759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.483726 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" event={"ID":"c4913026-0b4a-4a34-824b-e03e1a52fe40","Type":"ContainerStarted","Data":"4b59e71afe82450979b810d96aa7d4b13a8dbbfbfefa9ae4e59ba2cbcc3159f9"} Nov 03 20:20:58.483972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.483789 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:20:58.483972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.483806 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" event={"ID":"c4913026-0b4a-4a34-824b-e03e1a52fe40","Type":"ContainerStarted","Data":"49838d722b6e84ee3252e92f04f3a38c8d4fca9facfa2000753538324982cb86"} Nov 03 20:20:58.485489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.485458 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" event={"ID":"3e0d9e70-1046-4968-8f8a-b5abc530753a","Type":"ContainerStarted","Data":"f83951615cb153e7d5aab3b2274b767f2b5ff6e0a2142665ab7cca086c19730d"} Nov 03 20:20:58.485689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.485550 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" event={"ID":"3e0d9e70-1046-4968-8f8a-b5abc530753a","Type":"ContainerStarted","Data":"1c62043b64154ba39304afae196d7b73513066ca9ab01f286df292033d64e0cf"} Nov 03 20:20:58.485689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.485583 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:20:58.505826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.505781 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" podStartSLOduration=2.505768675 podStartE2EDuration="2.505768675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:58.501869799 +0000 UTC m=+1168.032215475" watchObservedRunningTime="2026-11-03 20:20:58.505768675 +0000 UTC m=+1168.036114374" Nov 03 20:20:58.527169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.527123 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" podStartSLOduration=2.527104952 podStartE2EDuration="2.527104952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:58.519893822 +0000 UTC m=+1168.050239496" watchObservedRunningTime="2026-11-03 20:20:58.527104952 +0000 UTC m=+1168.057450626" Nov 03 20:20:58.587638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:20:58.587608 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed949d1-a211-4bc3-8e1f-37dc2eeebefd" path="/var/lib/kubelet/pods/bed949d1-a211-4bc3-8e1f-37dc2eeebefd/volumes" Nov 03 20:21:01.648720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.648683 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2"] Nov 03 20:21:01.654076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.654053 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.669461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.669438 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2"] Nov 03 20:21:01.688832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.688808 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-400777-openshift-default-dockercfg-5vrgg\"" Nov 03 20:21:01.752125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752087 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752235 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752289 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzzl\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752378 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752425 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752545 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752637 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752670 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.752808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.752729 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.872535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.869259 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.872535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.869310 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.872535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.869467 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.872535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.869523 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.872535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.869567 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.894543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.892207 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.905596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.903545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.923543 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.923742 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.923754 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.923881 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.923941 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzzl\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.923979 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.924530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.924052 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.984551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.982132 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzzl\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.984551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.983025 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:01.994530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:01.985000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:02.043272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.043223 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo\") pod \"e2e-api-b-400777-openshift-default-845498b657-kg7b2\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:02.153544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.140752 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm"] Nov 03 20:21:02.192475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.192379 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg"] Nov 03 20:21:02.238622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.238577 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582fd919-dghcm"] Nov 03 20:21:02.328414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.309212 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:02.338923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.338887 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1abe20f-7b6cg"] Nov 03 20:21:02.608925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.606234 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d" path="/var/lib/kubelet/pods/3d194f4e-2b2b-4c54-a7e7-d44c8c4d7f9d/volumes" Nov 03 20:21:02.608925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.607071 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c5d4dd-992b-4d96-b86b-cb856d873bd6" path="/var/lib/kubelet/pods/a8c5d4dd-992b-4d96-b86b-cb856d873bd6/volumes" Nov 03 20:21:02.930897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.926619 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2"] Nov 03 20:21:02.962614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.962568 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:02.962941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.962917 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:02.963133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:02.963113 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:03.597461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:03.597337 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" event={"ID":"c18d069c-790a-483d-a6bd-78ce0e56216a","Type":"ContainerStarted","Data":"17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58"} Nov 03 20:21:03.597461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:03.597434 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" event={"ID":"c18d069c-790a-483d-a6bd-78ce0e56216a","Type":"ContainerStarted","Data":"c0d7d3cf0909938d29cb627ecbc4b5a188b8b6fd2fdd3f154909b7ef85c623fa"} Nov 03 20:21:03.627545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:03.627477 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podStartSLOduration=2.6274640529999997 podStartE2EDuration="2.627464053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:03.622761615 +0000 UTC m=+1173.153107290" watchObservedRunningTime="2026-11-03 20:21:03.627464053 +0000 UTC m=+1173.157809725" Nov 03 20:21:04.329433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:04.329393 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:04.335395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:04.335375 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:04.606024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:04.605956 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:04.606859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:04.606837 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:21:07.086591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.086550 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz"] Nov 03 20:21:07.091020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.090997 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.100788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.099113 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-400777\"" Nov 03 20:21:07.113618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.113594 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz"] Nov 03 20:21:07.114424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.114400 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh"] Nov 03 20:21:07.133191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.123484 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.133191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.132009 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-400777-dockercfg-8n5dm\"" Nov 03 20:21:07.151242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.151215 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh"] Nov 03 20:21:07.165255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.165221 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.165398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.165282 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume\") pod \"payload-processing-e2e-api-b-400777-776c774778-4mwtz\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.165398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.165389 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2fp\" (UniqueName: \"kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp\") pod \"payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.165527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.165464 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxlj\" (UniqueName: \"kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj\") pod \"payload-processing-e2e-api-b-400777-776c774778-4mwtz\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.267426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.267392 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2fp\" (UniqueName: \"kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp\") pod \"payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.267607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.267489 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxlj\" (UniqueName: \"kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj\") pod \"payload-processing-e2e-api-b-400777-776c774778-4mwtz\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.267710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.267692 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.267752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.267740 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume\") pod \"payload-processing-e2e-api-b-400777-776c774778-4mwtz\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.268776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.268734 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.268776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.268748 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume\") pod \"payload-processing-e2e-api-b-400777-776c774778-4mwtz\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.279985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.279958 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2fp\" (UniqueName: \"kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp\") pod \"payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.282111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.282084 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxlj\" (UniqueName: \"kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj\") pod \"payload-processing-e2e-api-b-400777-776c774778-4mwtz\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.427126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.426752 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:07.475100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.474887 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:07.885282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:07.885246 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz"] Nov 03 20:21:08.200064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.200043 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh"] Nov 03 20:21:08.705612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.702205 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" event={"ID":"1b5d4152-feea-4abe-9c91-e02cf17f544b","Type":"ContainerStarted","Data":"960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e"} Nov 03 20:21:08.705612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.702272 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" event={"ID":"1b5d4152-feea-4abe-9c91-e02cf17f544b","Type":"ContainerStarted","Data":"79f976cb3b92233d89eb1006ea24e7e5f78980094f8ef00882e8fd24505744df"} Nov 03 20:21:08.705612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.703384 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:08.712755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.707483 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" event={"ID":"51f7797d-f982-4357-a209-b26e04c9cd31","Type":"ContainerStarted","Data":"36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd"} Nov 03 20:21:08.712755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.707538 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" event={"ID":"51f7797d-f982-4357-a209-b26e04c9cd31","Type":"ContainerStarted","Data":"e0089100c77a119d4270178bcea0c7dda631424b0f779cf6df881d0799bc650b"} Nov 03 20:21:08.712755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.708053 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:08.733476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.733423 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" podStartSLOduration=1.733406058 podStartE2EDuration="1.733406058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:08.722339224 +0000 UTC m=+1178.252684926" watchObservedRunningTime="2026-11-03 20:21:08.733406058 +0000 UTC m=+1178.263751730" Nov 03 20:21:08.743875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:08.743836 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" podStartSLOduration=1.743824251 podStartE2EDuration="1.743824251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:08.742717983 +0000 UTC m=+1178.273063654" watchObservedRunningTime="2026-11-03 20:21:08.743824251 +0000 UTC m=+1178.274169934" Nov 03 20:21:09.028721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.028693 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t"] Nov 03 20:21:09.034796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.034771 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7228736-h8s9t"] Nov 03 20:21:09.527534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.527122 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:21:09.535887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.535865 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:21:09.845705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.845614 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk"] Nov 03 20:21:09.852954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.852932 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk"] Nov 03 20:21:09.853136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.853076 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927476 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbhs\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927588 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927628 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927756 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927783 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927878 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.927970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.927959 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.928069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.928056 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.928104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.928094 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:09.937189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:09.937171 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9b65143e-openshift-default-dockercfg-sd5gc\"" Nov 03 20:21:10.029202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029172 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029227 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029294 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029324 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029358 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbhs\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029387 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029580 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029697 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029735 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029739 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029848 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.029999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029978 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.030064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.029996 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.030064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.030043 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.032046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.032025 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.032247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.032230 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.043327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.043304 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.043419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.043353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbhs\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.169421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.169351 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:10.462558 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:21:10.462523 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494771ec_7bd4_4e96_8174_0685fe306e0e.slice/crio-f1fa71ecc74487affe99123c5c7b773093db54722c516f0a00fbcd5bfb8e088d WatchSource:0}: Error finding container f1fa71ecc74487affe99123c5c7b773093db54722c516f0a00fbcd5bfb8e088d: Status 404 returned error can't find the container with id f1fa71ecc74487affe99123c5c7b773093db54722c516f0a00fbcd5bfb8e088d Nov 03 20:21:10.463020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.462992 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk"] Nov 03 20:21:10.465146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.465113 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:10.465220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.465179 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:10.465220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.465213 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:10.594927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.594895 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80a7c99-f35f-4267-ba57-4b5049498ad1" path="/var/lib/kubelet/pods/d80a7c99-f35f-4267-ba57-4b5049498ad1/volumes" Nov 03 20:21:10.736638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.736582 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" event={"ID":"494771ec-7bd4-4e96-8174-0685fe306e0e","Type":"ContainerStarted","Data":"c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e"} Nov 03 20:21:10.736638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.736641 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" event={"ID":"494771ec-7bd4-4e96-8174-0685fe306e0e","Type":"ContainerStarted","Data":"f1fa71ecc74487affe99123c5c7b773093db54722c516f0a00fbcd5bfb8e088d"} Nov 03 20:21:10.763750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:10.760826 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podStartSLOduration=1.760809948 podStartE2EDuration="1.760809948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:10.758953703 +0000 UTC m=+1180.289299377" watchObservedRunningTime="2026-11-03 20:21:10.760809948 +0000 UTC m=+1180.291155620" Nov 03 20:21:11.180600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:11.177799 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:11.180600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:11.177935 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:11.180600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:11.177991 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:11.986571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:11.968171 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59"] Nov 03 20:21:11.991341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:11.991308 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h"] Nov 03 20:21:12.193533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:12.184959 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:12.193533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:12.185053 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:13.175556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:13.173873 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:13.175556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:13.173941 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:13.853741 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:21:13.853705 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:21:14.071533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:14.071073 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd"] Nov 03 20:21:14.082733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:14.082686 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6666e35-cpbbd"] Nov 03 20:21:14.176583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:14.169832 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:14.176583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:14.169889 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:14.619565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:14.617908 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519543cd-7ad0-42fd-ad76-4db07e5d6782" path="/var/lib/kubelet/pods/519543cd-7ad0-42fd-ad76-4db07e5d6782/volumes" Nov 03 20:21:15.029541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:15.029469 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs"] Nov 03 20:21:15.033836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:15.033811 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-436fd837-rczvs"] Nov 03 20:21:15.186535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:15.185622 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:15.186535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:15.185694 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:16.172932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:16.172893 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:16.173255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:16.173216 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:16.623927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:16.623890 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a604c0-02b9-45d6-b30c-718d37d5d0c7" path="/var/lib/kubelet/pods/a1a604c0-02b9-45d6-b30c-718d37d5d0c7/volumes" Nov 03 20:21:16.982568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:16.969597 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" podUID="fea1f570-0d09-4add-b1a0-6bf2bc1c053c" containerName="payload-processing" containerID="cri-o://32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584" gracePeriod=25 Nov 03 20:21:16.993657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:16.993597 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" podUID="e5a46758-59f5-434f-81dd-c01fda609e7a" containerName="payload-pre-processing" containerID="cri-o://9177cebb14ca653011ab21d27fb61bca3b5b92704d8c263fe21b88d48ab71dad" gracePeriod=25 Nov 03 20:21:17.188532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:17.180104 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:17.188532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:17.180170 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:18.047262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.047233 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:21:18.049307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.049192 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm"] Nov 03 20:21:18.051161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.050934 5815 generic.go:356] "Generic (PLEG): container finished" podID="fea1f570-0d09-4add-b1a0-6bf2bc1c053c" containerID="32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584" exitCode=0 Nov 03 20:21:18.051161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.051021 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" event={"ID":"fea1f570-0d09-4add-b1a0-6bf2bc1c053c","Type":"ContainerDied","Data":"32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584"} Nov 03 20:21:18.051161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.051094 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" event={"ID":"fea1f570-0d09-4add-b1a0-6bf2bc1c053c","Type":"ContainerDied","Data":"f7fdb0dce780eace12682057aa50c08e506f45cdb34bf5b4abeaff266f5010f3"} Nov 03 20:21:18.051161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.051124 5815 scope.go:122] "RemoveContainer" containerID="32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584" Nov 03 20:21:18.051436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.051259 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59" Nov 03 20:21:18.055626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.055601 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99718ec-89kdm"] Nov 03 20:21:18.055725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.055665 5815 generic.go:356] "Generic (PLEG): container finished" podID="e5a46758-59f5-434f-81dd-c01fda609e7a" containerID="9177cebb14ca653011ab21d27fb61bca3b5b92704d8c263fe21b88d48ab71dad" exitCode=0 Nov 03 20:21:18.056543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.056484 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" event={"ID":"e5a46758-59f5-434f-81dd-c01fda609e7a","Type":"ContainerDied","Data":"9177cebb14ca653011ab21d27fb61bca3b5b92704d8c263fe21b88d48ab71dad"} Nov 03 20:21:18.115779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.115722 5815 scope.go:122] "RemoveContainer" containerID="32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584" Nov 03 20:21:18.118622 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:21:18.116863 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584\": container with ID starting with 32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584 not found: ID does not exist" containerID="32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584" Nov 03 20:21:18.119223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.119194 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584"} err="failed to get container status \"32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584\": rpc error: code = NotFound desc = could not find container \"32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584\": container with ID starting with 32ef71b5a5964b172dd6c43dd0059eb58ab4be165a70783eff2c22d149394584 not found: ID does not exist" Nov 03 20:21:18.148945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.148918 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz\" (UniqueName: \"kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz\") pod \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " Nov 03 20:21:18.149099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.149023 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume\") pod \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\" (UID: \"fea1f570-0d09-4add-b1a0-6bf2bc1c053c\") " Nov 03 20:21:18.149377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.149353 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume" pod "fea1f570-0d09-4add-b1a0-6bf2bc1c053c" (UID: "fea1f570-0d09-4add-b1a0-6bf2bc1c053c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:18.149629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.149611 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:18.151452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.151427 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz" pod "fea1f570-0d09-4add-b1a0-6bf2bc1c053c" (UID: "fea1f570-0d09-4add-b1a0-6bf2bc1c053c"). InnerVolumeSpecName "kube-api-access-8jvbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:18.181531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.177964 5815 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:21:18.181531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.178027 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:21:18.224532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.219150 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:21:18.250731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.250704 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume\") pod \"e5a46758-59f5-434f-81dd-c01fda609e7a\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " Nov 03 20:21:18.250896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.250824 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv\" (UniqueName: \"kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv\") pod \"e5a46758-59f5-434f-81dd-c01fda609e7a\" (UID: \"e5a46758-59f5-434f-81dd-c01fda609e7a\") " Nov 03 20:21:18.251054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.251034 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume" pod "e5a46758-59f5-434f-81dd-c01fda609e7a" (UID: "e5a46758-59f5-434f-81dd-c01fda609e7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:18.251336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.251315 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5a46758-59f5-434f-81dd-c01fda609e7a-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:18.251473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.251338 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvbz\" (UniqueName: \"kubernetes.io/projected/fea1f570-0d09-4add-b1a0-6bf2bc1c053c-kube-api-access-8jvbz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:18.253709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.253687 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv" pod "e5a46758-59f5-434f-81dd-c01fda609e7a" (UID: "e5a46758-59f5-434f-81dd-c01fda609e7a"). InnerVolumeSpecName "kube-api-access-5jhtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:18.352883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.352854 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jhtv\" (UniqueName: \"kubernetes.io/projected/e5a46758-59f5-434f-81dd-c01fda609e7a-kube-api-access-5jhtv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:18.413840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.413754 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59"] Nov 03 20:21:18.418598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.418574 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-vns59"] Nov 03 20:21:18.613259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.613222 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5258391-537b-490d-bde5-9afd62127221" path="/var/lib/kubelet/pods/a5258391-537b-490d-bde5-9afd62127221/volumes" Nov 03 20:21:18.613893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:18.613873 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea1f570-0d09-4add-b1a0-6bf2bc1c053c" path="/var/lib/kubelet/pods/fea1f570-0d09-4add-b1a0-6bf2bc1c053c/volumes" Nov 03 20:21:19.073993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.073935 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" event={"ID":"e5a46758-59f5-434f-81dd-c01fda609e7a","Type":"ContainerDied","Data":"6b8ccce8b016148d4e355d0e7f3e2e7d89426de199b47be0d040d67135ca116b"} Nov 03 20:21:19.074555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.074012 5815 scope.go:122] "RemoveContainer" containerID="9177cebb14ca653011ab21d27fb61bca3b5b92704d8c263fe21b88d48ab71dad" Nov 03 20:21:19.074555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.074121 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h" Nov 03 20:21:19.122585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.117529 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h"] Nov 03 20:21:19.129763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.129729 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-zkr2h"] Nov 03 20:21:19.525671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.525643 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:19.526038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.526015 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:19.526756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.526737 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:19.724544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:19.722873 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:21:20.718532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:20.704082 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5a46758-59f5-434f-81dd-c01fda609e7a" path="/var/lib/kubelet/pods/e5a46758-59f5-434f-81dd-c01fda609e7a/volumes" Nov 03 20:21:20.764537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:20.763131 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:21:23.029201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:23.029122 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr"] Nov 03 20:21:23.033427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:23.033397 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd72ef1e-lrlrr"] Nov 03 20:21:24.589466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.589433 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f1b536-89ed-44db-95b9-0bb53ae16b51" path="/var/lib/kubelet/pods/40f1b536-89ed-44db-95b9-0bb53ae16b51/volumes" Nov 03 20:21:24.888318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.888235 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr"] Nov 03 20:21:24.889317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.889293 5815 cpu_manager.go:414] "containerMap: removing container" podUID="fea1f570-0d09-4add-b1a0-6bf2bc1c053c" containerName="payload-processing" Nov 03 20:21:24.889448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.889319 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea1f570-0d09-4add-b1a0-6bf2bc1c053c" containerName="payload-processing" Nov 03 20:21:24.889448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.889377 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e5a46758-59f5-434f-81dd-c01fda609e7a" containerName="payload-pre-processing" Nov 03 20:21:24.889448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.889386 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5a46758-59f5-434f-81dd-c01fda609e7a" containerName="payload-pre-processing" Nov 03 20:21:24.889692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.889594 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea1f570-0d09-4add-b1a0-6bf2bc1c053c" containerName="payload-processing" Nov 03 20:21:24.889692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.889610 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5a46758-59f5-434f-81dd-c01fda609e7a" containerName="payload-pre-processing" Nov 03 20:21:24.897207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.896126 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:24.909459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.909418 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b07e26-dockercfg-z92n5\"" Nov 03 20:21:24.916712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.916683 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr"] Nov 03 20:21:24.965209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:24.965161 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b07e26\"" Nov 03 20:21:25.097019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.092748 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.097019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.093001 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5rs\" (UniqueName: \"kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.196522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.196421 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5rs\" (UniqueName: \"kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.196742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.196707 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.197780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.197751 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.244257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.244178 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5rs\" (UniqueName: \"kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.539597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.532026 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:25.893477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.893393 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5"] Nov 03 20:21:25.913274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.913233 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv"] Nov 03 20:21:25.956429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:25.956393 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr"] Nov 03 20:21:26.364235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:26.362812 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" event={"ID":"ffba69fd-250c-4f60-bbc3-6278de6bace0","Type":"ContainerStarted","Data":"88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1"} Nov 03 20:21:26.364235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:26.362884 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" event={"ID":"ffba69fd-250c-4f60-bbc3-6278de6bace0","Type":"ContainerStarted","Data":"b1d6960aaa954c617606e3203a57ebcef399a07f8790927cda88c1c1980fcb7d"} Nov 03 20:21:26.364235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:26.364128 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:26.429532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:26.422696 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" podStartSLOduration=2.422672396 podStartE2EDuration="2.422672396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:26.396871433 +0000 UTC m=+1195.927217106" watchObservedRunningTime="2026-11-03 20:21:26.422672396 +0000 UTC m=+1195.953018068" Nov 03 20:21:26.503737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:26.503700 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk"] Nov 03 20:21:26.504231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:26.504195 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" containerID="cri-o://c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e" gracePeriod=30 Nov 03 20:21:29.043770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.043703 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88"] Nov 03 20:21:29.079778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.079735 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c214eae-vfr88"] Nov 03 20:21:29.421237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.421141 5815 generic.go:356] "Generic (PLEG): container finished" podID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerID="09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162" exitCode=0 Nov 03 20:21:29.421403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.421259 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" event={"ID":"bfb070e8-cb4e-467b-b35d-5c8b0a77e475","Type":"ContainerDied","Data":"09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162"} Nov 03 20:21:29.421893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.421868 5815 scope.go:122] "RemoveContainer" containerID="09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162" Nov 03 20:21:29.424013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.423976 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:29.424128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.424071 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:29.424128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:29.424121 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:30.449998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.449967 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" event={"ID":"bfb070e8-cb4e-467b-b35d-5c8b0a77e475","Type":"ContainerStarted","Data":"8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743"} Nov 03 20:21:30.606965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.605750 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:21:30.606965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.605813 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:21:30.619690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.619645 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5002cd39-5bd6-4d7a-929c-6787ca6c5754" path="/var/lib/kubelet/pods/5002cd39-5bd6-4d7a-929c-6787ca6c5754/volumes" Nov 03 20:21:30.620146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.620125 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:21:30.896338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.895581 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" podUID="3e0d9e70-1046-4968-8f8a-b5abc530753a" containerName="payload-processing" containerID="cri-o://f83951615cb153e7d5aab3b2274b767f2b5ff6e0a2142665ab7cca086c19730d" gracePeriod=25 Nov 03 20:21:30.913947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:30.913903 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" podUID="c4913026-0b4a-4a34-824b-e03e1a52fe40" containerName="payload-pre-processing" containerID="cri-o://4b59e71afe82450979b810d96aa7d4b13a8dbbfbfefa9ae4e59ba2cbcc3159f9" gracePeriod=25 Nov 03 20:21:31.032555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.029067 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9"] Nov 03 20:21:31.039479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.039455 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a775fabb-krxv9"] Nov 03 20:21:31.513158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.512139 5815 generic.go:356] "Generic (PLEG): container finished" podID="3e0d9e70-1046-4968-8f8a-b5abc530753a" containerID="f83951615cb153e7d5aab3b2274b767f2b5ff6e0a2142665ab7cca086c19730d" exitCode=0 Nov 03 20:21:31.513158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.512476 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" event={"ID":"3e0d9e70-1046-4968-8f8a-b5abc530753a","Type":"ContainerDied","Data":"f83951615cb153e7d5aab3b2274b767f2b5ff6e0a2142665ab7cca086c19730d"} Nov 03 20:21:31.513158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.512615 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" event={"ID":"3e0d9e70-1046-4968-8f8a-b5abc530753a","Type":"ContainerDied","Data":"1c62043b64154ba39304afae196d7b73513066ca9ab01f286df292033d64e0cf"} Nov 03 20:21:31.513158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.512647 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c62043b64154ba39304afae196d7b73513066ca9ab01f286df292033d64e0cf" Nov 03 20:21:31.517552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.517080 5815 generic.go:356] "Generic (PLEG): container finished" podID="c4913026-0b4a-4a34-824b-e03e1a52fe40" containerID="4b59e71afe82450979b810d96aa7d4b13a8dbbfbfefa9ae4e59ba2cbcc3159f9" exitCode=0 Nov 03 20:21:31.517670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.517572 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" event={"ID":"c4913026-0b4a-4a34-824b-e03e1a52fe40","Type":"ContainerDied","Data":"4b59e71afe82450979b810d96aa7d4b13a8dbbfbfefa9ae4e59ba2cbcc3159f9"} Nov 03 20:21:31.524292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.524267 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:21:31.528003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.527972 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:21:31.611708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.611673 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:21:31.611854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.611733 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:21:31.637203 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:21:31.637178 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494771ec_7bd4_4e96_8174_0685fe306e0e.slice/crio-f1fa71ecc74487affe99123c5c7b773093db54722c516f0a00fbcd5bfb8e088d\": RecentStats: unable to find data in memory cache]" Nov 03 20:21:31.659080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.659053 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume\") pod \"3e0d9e70-1046-4968-8f8a-b5abc530753a\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " Nov 03 20:21:31.659206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.659152 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume\") pod \"c4913026-0b4a-4a34-824b-e03e1a52fe40\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " Nov 03 20:21:31.659255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.659241 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9\" (UniqueName: \"kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9\") pod \"3e0d9e70-1046-4968-8f8a-b5abc530753a\" (UID: \"3e0d9e70-1046-4968-8f8a-b5abc530753a\") " Nov 03 20:21:31.659291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.659281 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9\" (UniqueName: \"kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9\") pod \"c4913026-0b4a-4a34-824b-e03e1a52fe40\" (UID: \"c4913026-0b4a-4a34-824b-e03e1a52fe40\") " Nov 03 20:21:31.660869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.660845 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume" pod "c4913026-0b4a-4a34-824b-e03e1a52fe40" (UID: "c4913026-0b4a-4a34-824b-e03e1a52fe40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:31.661161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.661144 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume" pod "3e0d9e70-1046-4968-8f8a-b5abc530753a" (UID: "3e0d9e70-1046-4968-8f8a-b5abc530753a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:31.663414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.663384 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9" pod "3e0d9e70-1046-4968-8f8a-b5abc530753a" (UID: "3e0d9e70-1046-4968-8f8a-b5abc530753a"). InnerVolumeSpecName "kube-api-access-f9rw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:31.688531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.677588 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9" pod "c4913026-0b4a-4a34-824b-e03e1a52fe40" (UID: "c4913026-0b4a-4a34-824b-e03e1a52fe40"). InnerVolumeSpecName "kube-api-access-hfhn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:31.763021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.761397 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rw9\" (UniqueName: \"kubernetes.io/projected/3e0d9e70-1046-4968-8f8a-b5abc530753a-kube-api-access-f9rw9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.763021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.761436 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhn9\" (UniqueName: \"kubernetes.io/projected/c4913026-0b4a-4a34-824b-e03e1a52fe40-kube-api-access-hfhn9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.763021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.761449 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0d9e70-1046-4968-8f8a-b5abc530753a-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.763021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:31.761460 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4913026-0b4a-4a34-824b-e03e1a52fe40-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.011232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.011203 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:32.026268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026210 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs"] Nov 03 20:21:32.026909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026893 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3e0d9e70-1046-4968-8f8a-b5abc530753a" containerName="payload-processing" Nov 03 20:21:32.026958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026911 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0d9e70-1046-4968-8f8a-b5abc530753a" containerName="payload-processing" Nov 03 20:21:32.026958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026924 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c4913026-0b4a-4a34-824b-e03e1a52fe40" containerName="payload-pre-processing" Nov 03 20:21:32.026958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026930 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4913026-0b4a-4a34-824b-e03e1a52fe40" containerName="payload-pre-processing" Nov 03 20:21:32.026958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026952 5815 cpu_manager.go:414] "containerMap: removing container" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" Nov 03 20:21:32.026958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.026957 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" Nov 03 20:21:32.027125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.027061 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0d9e70-1046-4968-8f8a-b5abc530753a" containerName="payload-processing" Nov 03 20:21:32.027125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.027070 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4913026-0b4a-4a34-824b-e03e1a52fe40" containerName="payload-pre-processing" Nov 03 20:21:32.027125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.027078 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerName="istio-proxy" Nov 03 20:21:32.029702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.029688 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.039806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.039759 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d6789f-dockercfg-dlb9l\"" Nov 03 20:21:32.045380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.044624 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d6789f\"" Nov 03 20:21:32.053129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.053102 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs"] Nov 03 20:21:32.064491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.064467 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9"] Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.065384 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.071996 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.076732 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.065496 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.084701 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.084745 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.084839 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.084902 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.084937 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.084996 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.085037 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token\") pod \"494771ec-7bd4-4e96-8174-0685fe306e0e\" (UID: \"494771ec-7bd4-4e96-8174-0685fe306e0e\") " Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.085341 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xtp\" (UniqueName: \"kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.085422 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.087536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.085858 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494771ec-7bd4-4e96-8174-0685fe306e0e-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.094925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.094742 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:32.126538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.107614 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:32.218735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.137610 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:32.218735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.157835 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:32.218735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.165693 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:32.218735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.189372 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xtp\" (UniqueName: \"kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.218735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.189434 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlkw\" (UniqueName: \"kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw\") pod \"payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.218735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.194381 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.189498 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.220837 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.221126 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.221144 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.221159 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.221173 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.227634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.221187 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.242529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.232322 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:32.242529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.239650 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "kube-api-access-8qbhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:32.242529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.239744 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy" pod "494771ec-7bd4-4e96-8174-0685fe306e0e" (UID: "494771ec-7bd4-4e96-8174-0685fe306e0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:32.328798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.327813 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlkw\" (UniqueName: \"kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw\") pod \"payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.328798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.327991 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.328798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.328323 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.328798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.328343 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qbhs\" (UniqueName: \"kubernetes.io/projected/494771ec-7bd4-4e96-8174-0685fe306e0e-kube-api-access-8qbhs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.328798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.328359 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494771ec-7bd4-4e96-8174-0685fe306e0e-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.331351 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.330501 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.343402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.332133 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xtp\" (UniqueName: \"kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.350302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.350040 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:32.449818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.434283 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlkw\" (UniqueName: \"kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw\") pod \"payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.451365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.451335 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9"] Nov 03 20:21:32.545468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.545441 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" event={"ID":"c4913026-0b4a-4a34-824b-e03e1a52fe40","Type":"ContainerDied","Data":"49838d722b6e84ee3252e92f04f3a38c8d4fca9facfa2000753538324982cb86"} Nov 03 20:21:32.545907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.545604 5815 scope.go:122] "RemoveContainer" containerID="4b59e71afe82450979b810d96aa7d4b13a8dbbfbfefa9ae4e59ba2cbcc3159f9" Nov 03 20:21:32.545907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.545458 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv" Nov 03 20:21:32.548028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.547979 5815 generic.go:356] "Generic (PLEG): container finished" podID="494771ec-7bd4-4e96-8174-0685fe306e0e" containerID="c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e" exitCode=0 Nov 03 20:21:32.548137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.548028 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" Nov 03 20:21:32.548216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.548138 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" event={"ID":"494771ec-7bd4-4e96-8174-0685fe306e0e","Type":"ContainerDied","Data":"c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e"} Nov 03 20:21:32.548216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.548182 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk" event={"ID":"494771ec-7bd4-4e96-8174-0685fe306e0e","Type":"ContainerDied","Data":"f1fa71ecc74487affe99123c5c7b773093db54722c516f0a00fbcd5bfb8e088d"} Nov 03 20:21:32.548353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.548313 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5" Nov 03 20:21:32.623378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.622935 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:21:32.623378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.622991 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:21:32.627000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.625077 5815 scope.go:122] "RemoveContainer" containerID="c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e" Nov 03 20:21:32.636359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.636337 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d7bbbae-6a84-49a8-a59b-771e82999294" path="/var/lib/kubelet/pods/4d7bbbae-6a84-49a8-a59b-771e82999294/volumes" Nov 03 20:21:32.651648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.643330 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv"] Nov 03 20:21:32.651648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.649718 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-759b9dfff9-vs2wv"] Nov 03 20:21:32.652829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.652809 5815 scope.go:122] "RemoveContainer" containerID="c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e" Nov 03 20:21:32.653187 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:21:32.653163 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e\": container with ID starting with c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e not found: ID does not exist" containerID="c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e" Nov 03 20:21:32.653273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.653201 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e"} err="failed to get container status \"c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e\": rpc error: code = NotFound desc = could not find container \"c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e\": container with ID starting with c5d9c7cb57e736614c0f2e0c5f8a2567303a9cd3aba1c83c781385ca0ac5f68e not found: ID does not exist" Nov 03 20:21:32.657404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.657350 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5"] Nov 03 20:21:32.708566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.702778 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:32.708566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.707455 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-595ccbdd5b-rjzt5"] Nov 03 20:21:32.755287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.754346 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk"] Nov 03 20:21:32.761418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.761397 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-kwbtk"] Nov 03 20:21:32.818674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.818636 5815 scope.go:122] "RemoveContainer" containerID="cd7627fc80254aac1ab64f073d7a88b4f2050cd7f861fd839d082d5356181834" Nov 03 20:21:32.898742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:32.898153 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs"] Nov 03 20:21:33.104078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.103262 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9"] Nov 03 20:21:33.558481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.558439 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" event={"ID":"d38a226e-b77f-4821-8f2c-87681e3830e0","Type":"ContainerStarted","Data":"92d686becb2268929e5a961535a8092883d7b3781ed7fb9564273e5bfbf9f73a"} Nov 03 20:21:33.558481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.558485 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" event={"ID":"d38a226e-b77f-4821-8f2c-87681e3830e0","Type":"ContainerStarted","Data":"4c7cd0ca1e561c7b49395924c3fc173e0eedf99f842f95432ca1e1f4aa38e31b"} Nov 03 20:21:33.559039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.558581 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:33.560352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.560323 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" event={"ID":"d70ed4b2-8a73-432d-b12e-7c9e1a85c966","Type":"ContainerStarted","Data":"cf790de80d04d9a61ed9bc830103e68e04294f930b5fe037b6e22ee2193cd121"} Nov 03 20:21:33.560468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.560376 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:33.560468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.560401 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" event={"ID":"d70ed4b2-8a73-432d-b12e-7c9e1a85c966","Type":"ContainerStarted","Data":"e432346cb519e0024c11739e72927980b96b57ff96beefcdecaa679fb30e56eb"} Nov 03 20:21:33.599529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.595075 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" podStartSLOduration=1.5950579139999999 podStartE2EDuration="1.595057914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:33.576805383 +0000 UTC m=+1203.107151059" watchObservedRunningTime="2026-11-03 20:21:33.595057914 +0000 UTC m=+1203.125403587" Nov 03 20:21:33.607536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.605188 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:21:33.607536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.605239 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:21:33.607536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.607054 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" podStartSLOduration=1.607039197 podStartE2EDuration="1.607039197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:33.603831289 +0000 UTC m=+1203.134176962" watchObservedRunningTime="2026-11-03 20:21:33.607039197 +0000 UTC m=+1203.137384875" Nov 03 20:21:33.957612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:33.957044 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456"] Nov 03 20:21:34.196774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.196737 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456"] Nov 03 20:21:34.198365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.198265 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.222250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.222223 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2eee8c\"" Nov 03 20:21:34.223267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.222800 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2eee8c-dockercfg-5fwd9\"" Nov 03 20:21:34.342997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.342960 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.417140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.417106 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfwv\" (UniqueName: \"kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.443059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.392599 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks"] Nov 03 20:21:34.453916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.453888 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks"] Nov 03 20:21:34.454085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.454007 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.543178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.543103 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:21:34.544078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.544037 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.544221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.544107 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.544290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.544223 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfwv\" (UniqueName: \"kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.544371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.544354 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vp8\" (UniqueName: \"kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8\") pod \"payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.544964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.544940 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.547919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.547905 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:21:34.551672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.551660 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:21:34.556319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.556300 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:21:34.557983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.557968 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfwv\" (UniqueName: \"kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.618534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.615223 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:21:34.618534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.615292 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:21:34.726529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.714115 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.726529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.714418 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vp8\" (UniqueName: \"kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8\") pod \"payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.726529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.716256 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.742169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.742127 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vp8\" (UniqueName: \"kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8\") pod \"payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.742849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.742810 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:34.791536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.788635 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0d9e70-1046-4968-8f8a-b5abc530753a" path="/var/lib/kubelet/pods/3e0d9e70-1046-4968-8f8a-b5abc530753a/volumes" Nov 03 20:21:34.813845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.792799 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:34.820429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.815096 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494771ec-7bd4-4e96-8174-0685fe306e0e" path="/var/lib/kubelet/pods/494771ec-7bd4-4e96-8174-0685fe306e0e/volumes" Nov 03 20:21:34.820429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:34.818474 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4913026-0b4a-4a34-824b-e03e1a52fe40" path="/var/lib/kubelet/pods/c4913026-0b4a-4a34-824b-e03e1a52fe40/volumes" Nov 03 20:21:35.194556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.192678 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456"] Nov 03 20:21:35.209774 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:21:35.209738 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ffaecf_165c_4ca9_9a70_8f416f8e0997.slice/crio-62f6bf52376e03b9685d7d38e0702bc3d222d2f9aadcf00e3826510f34581ac7 WatchSource:0}: Error finding container 62f6bf52376e03b9685d7d38e0702bc3d222d2f9aadcf00e3826510f34581ac7: Status 404 returned error can't find the container with id 62f6bf52376e03b9685d7d38e0702bc3d222d2f9aadcf00e3826510f34581ac7 Nov 03 20:21:35.220992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.220968 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks"] Nov 03 20:21:35.599318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.599287 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" event={"ID":"74ffaecf-165c-4ca9-9a70-8f416f8e0997","Type":"ContainerStarted","Data":"7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76"} Nov 03 20:21:35.601644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.601623 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:35.601836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.601819 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" event={"ID":"74ffaecf-165c-4ca9-9a70-8f416f8e0997","Type":"ContainerStarted","Data":"62f6bf52376e03b9685d7d38e0702bc3d222d2f9aadcf00e3826510f34581ac7"} Nov 03 20:21:35.603153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.602611 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" event={"ID":"d24ef639-e473-4fad-9e56-af609646954b","Type":"ContainerStarted","Data":"a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f"} Nov 03 20:21:35.603153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.602660 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" event={"ID":"d24ef639-e473-4fad-9e56-af609646954b","Type":"ContainerStarted","Data":"f54208fa20411c51eb79aebca4c14acabba2920ff702a55deea25eada1bdb30e"} Nov 03 20:21:35.603153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.603066 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:35.606158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.606135 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:21:35.606328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.606307 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:21:35.622609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.622572 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" podStartSLOduration=2.622560359 podStartE2EDuration="2.622560359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:35.618901477 +0000 UTC m=+1205.149247151" watchObservedRunningTime="2026-11-03 20:21:35.622560359 +0000 UTC m=+1205.152906030" Nov 03 20:21:35.637965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:35.637920 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" podStartSLOduration=2.637907421 podStartE2EDuration="2.637907421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:35.637218123 +0000 UTC m=+1205.167563796" watchObservedRunningTime="2026-11-03 20:21:35.637907421 +0000 UTC m=+1205.168253093" Nov 03 20:21:36.271351 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:36.269254 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:21:36.613000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:36.612912 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:21:36.617004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:36.616976 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:21:37.025818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:37.025788 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g"] Nov 03 20:21:37.030038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:37.030013 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19b3dcb6-8g25g"] Nov 03 20:21:38.411382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:38.411355 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:21:38.595191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:38.595157 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e782f559-d001-4cf3-8361-5bc4ba3fe455" path="/var/lib/kubelet/pods/e782f559-d001-4cf3-8361-5bc4ba3fe455/volumes" Nov 03 20:21:38.980977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:38.980941 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt"] Nov 03 20:21:38.985896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:38.985874 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:38.994762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:38.994737 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt"] Nov 03 20:21:38.996175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:38.996153 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf9b0a\"" Nov 03 20:21:39.010131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.010108 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7"] Nov 03 20:21:39.014374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.014352 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.020797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.020013 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf9b0a-dockercfg-qbvkd\"" Nov 03 20:21:39.027989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.027970 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7"] Nov 03 20:21:39.116595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.116558 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2l4d\" (UniqueName: \"kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.116774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.116747 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.116841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.116784 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.116938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.116922 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g872\" (UniqueName: \"kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872\") pod \"payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.218632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.218564 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2l4d\" (UniqueName: \"kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.218845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.218780 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.218845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.218816 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.219193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.218944 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g872\" (UniqueName: \"kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872\") pod \"payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.234633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.220648 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.234633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.221479 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.244530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.241591 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g872\" (UniqueName: \"kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872\") pod \"payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.261859 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2l4d\" (UniqueName: \"kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d\") pod \"payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.327534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.326769 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:39.335585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.332847 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:39.830841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.830815 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7"] Nov 03 20:21:39.847443 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:21:39.847421 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bd15bf_a127_48be_8c0d_3827037a8a06.slice/crio-ce48e872681fd83e3f3425ae6bce8d25f251332af3463b7d0ccc2b9006d10f1d WatchSource:0}: Error finding container ce48e872681fd83e3f3425ae6bce8d25f251332af3463b7d0ccc2b9006d10f1d: Status 404 returned error can't find the container with id ce48e872681fd83e3f3425ae6bce8d25f251332af3463b7d0ccc2b9006d10f1d Nov 03 20:21:39.847933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:39.847909 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt"] Nov 03 20:21:40.654554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.654524 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" event={"ID":"e2bd15bf-a127-48be-8c0d-3827037a8a06","Type":"ContainerStarted","Data":"427d719f7d68a02ff43e4e1661ae3cedd820aa94c9263c316ba6de9744e4ad8f"} Nov 03 20:21:40.654746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.654573 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" event={"ID":"e2bd15bf-a127-48be-8c0d-3827037a8a06","Type":"ContainerStarted","Data":"ce48e872681fd83e3f3425ae6bce8d25f251332af3463b7d0ccc2b9006d10f1d"} Nov 03 20:21:40.654746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.654651 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:40.656395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.656367 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" event={"ID":"20ab9900-6823-467f-b78d-c332912d7c19","Type":"ContainerStarted","Data":"e549d5734b68742f959427d3e87ffebded0473ec9063debdce8c331674590d6c"} Nov 03 20:21:40.656559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.656408 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" event={"ID":"20ab9900-6823-467f-b78d-c332912d7c19","Type":"ContainerStarted","Data":"4e82ee7f999d4a5c35816ae6981e05dd300e962963654100075a5cb552afbd6f"} Nov 03 20:21:40.656559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.656430 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:40.705618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:40.705565 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" podStartSLOduration=2.705548868 podStartE2EDuration="2.705548868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:40.676076702 +0000 UTC m=+1210.206422375" watchObservedRunningTime="2026-11-03 20:21:40.705548868 +0000 UTC m=+1210.235894541" Nov 03 20:21:42.029168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:42.029116 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" podStartSLOduration=4.029101097 podStartE2EDuration="4.029101097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:40.706901207 +0000 UTC m=+1210.237246879" watchObservedRunningTime="2026-11-03 20:21:42.029101097 +0000 UTC m=+1211.559446798" Nov 03 20:21:42.030403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:42.030381 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6"] Nov 03 20:21:42.034486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:42.034464 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d195641-5gml6"] Nov 03 20:21:42.592921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:42.592893 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c92839-cf49-48b2-a24d-c984020d8393" path="/var/lib/kubelet/pods/b2c92839-cf49-48b2-a24d-c984020d8393/volumes" Nov 03 20:21:43.853378 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:21:43.853343 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:21:44.268234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:44.268197 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:21:44.599970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:44.599876 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:21:44.600151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:44.600013 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:21:46.642534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:46.642445 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:21:47.089596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:47.084565 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn"] Nov 03 20:21:47.176529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:47.175595 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28d919b-jddqn"] Nov 03 20:21:47.635352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:47.635190 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:21:48.635886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:48.635839 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3538e1a-50db-4421-a7bd-460e41b153c9" path="/var/lib/kubelet/pods/e3538e1a-50db-4421-a7bd-460e41b153c9/volumes" Nov 03 20:21:51.708553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:51.692671 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:21:51.708553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:51.704358 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:21:55.024164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:55.024135 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8"] Nov 03 20:21:55.033287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:55.030009 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a56be763-vzbt8"] Nov 03 20:21:55.424120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:55.423671 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:21:55.424120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:55.423754 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:56.590782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:56.590740 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca" path="/var/lib/kubelet/pods/5a4cb5c5-f7c7-436c-b43f-cd41cb0678ca/volumes" Nov 03 20:21:58.438724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.438644 5815 generic.go:356] "Generic (PLEG): container finished" podID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerID="17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58" exitCode=0 Nov 03 20:21:58.439235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.438733 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" event={"ID":"c18d069c-790a-483d-a6bd-78ce0e56216a","Type":"ContainerDied","Data":"17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58"} Nov 03 20:21:58.439235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.439205 5815 scope.go:122] "RemoveContainer" containerID="17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58" Nov 03 20:21:58.441046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.441025 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerID="9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f" exitCode=0 Nov 03 20:21:58.441152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.441095 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" event={"ID":"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321","Type":"ContainerDied","Data":"9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f"} Nov 03 20:21:58.441152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.441113 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:58.441266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.441194 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:58.441266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.441228 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:58.441482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.441466 5815 scope.go:122] "RemoveContainer" containerID="9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f" Nov 03 20:21:58.452555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.443679 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:58.452555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.443875 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:58.452555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:58.443981 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:21:59.036667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:59.036636 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7"] Nov 03 20:21:59.056734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:59.051893 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf931bb-9zkq7"] Nov 03 20:21:59.558179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:59.558135 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" event={"ID":"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321","Type":"ContainerStarted","Data":"7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7"} Nov 03 20:21:59.562226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:21:59.562171 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" event={"ID":"c18d069c-790a-483d-a6bd-78ce0e56216a","Type":"ContainerStarted","Data":"2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a"} Nov 03 20:22:00.102931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.094191 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:22:00.102931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.094383 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.102931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.094431 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:00.329819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.329571 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:22:00.330051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.330017 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:00.330335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.330245 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:00.699860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:00.699827 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a22b04-cb03-4be6-a415-536e9cecac10" path="/var/lib/kubelet/pods/54a22b04-cb03-4be6-a415-536e9cecac10/volumes" Nov 03 20:22:01.102940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.102899 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.103231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.103204 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:01.330027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.329994 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:01.330266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.330245 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:01.544829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.544792 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq"] Nov 03 20:22:01.545631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.545336 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" podUID="5c797e9f-208a-4378-97c8-78a93156f86a" containerName="payload-processing" containerID="cri-o://baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356" gracePeriod=30 Nov 03 20:22:01.828154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.827609 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9"] Nov 03 20:22:01.828154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:01.827876 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" podUID="a07f22ff-a809-4060-a9d8-74944518d18e" containerName="payload-pre-processing" containerID="cri-o://cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b" gracePeriod=30 Nov 03 20:22:02.113619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.108826 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:02.113619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.108885 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:02.331164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.330640 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:02.331164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.330717 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:02.785563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.774330 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:22:02.843531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.829477 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks"] Nov 03 20:22:02.888978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.871736 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456"] Nov 03 20:22:02.934532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.922842 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume\") pod \"5c797e9f-208a-4378-97c8-78a93156f86a\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " Nov 03 20:22:02.934532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.923017 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v\" (UniqueName: \"kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v\") pod \"5c797e9f-208a-4378-97c8-78a93156f86a\" (UID: \"5c797e9f-208a-4378-97c8-78a93156f86a\") " Nov 03 20:22:02.952540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.943837 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume" pod "5c797e9f-208a-4378-97c8-78a93156f86a" (UID: "5c797e9f-208a-4378-97c8-78a93156f86a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:02.958799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:02.958767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v" pod "5c797e9f-208a-4378-97c8-78a93156f86a" (UID: "5c797e9f-208a-4378-97c8-78a93156f86a"). InnerVolumeSpecName "kube-api-access-nbj8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:03.036641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.025719 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c797e9f-208a-4378-97c8-78a93156f86a-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.036979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.036950 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbj8v\" (UniqueName: \"kubernetes.io/projected/5c797e9f-208a-4378-97c8-78a93156f86a-kube-api-access-nbj8v\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.100419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.100387 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:03.100708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.100678 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:03.245004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.244970 5815 generic.go:356] "Generic (PLEG): container finished" podID="5c797e9f-208a-4378-97c8-78a93156f86a" containerID="baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356" exitCode=0 Nov 03 20:22:03.245303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.245284 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" event={"ID":"5c797e9f-208a-4378-97c8-78a93156f86a","Type":"ContainerDied","Data":"baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356"} Nov 03 20:22:03.245462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.245449 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" event={"ID":"5c797e9f-208a-4378-97c8-78a93156f86a","Type":"ContainerDied","Data":"49f7bc0f47e45832f10bcf8bcf2e61a8f676a3a3156a71394ec4e0e4082e3328"} Nov 03 20:22:03.245604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.245592 5815 scope.go:122] "RemoveContainer" containerID="baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356" Nov 03 20:22:03.245858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.245843 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq" Nov 03 20:22:03.246206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.246190 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:22:03.269240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.269203 5815 generic.go:356] "Generic (PLEG): container finished" podID="a07f22ff-a809-4060-a9d8-74944518d18e" containerID="cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b" exitCode=0 Nov 03 20:22:03.305813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.269578 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" event={"ID":"a07f22ff-a809-4060-a9d8-74944518d18e","Type":"ContainerDied","Data":"cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b"} Nov 03 20:22:03.306110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.306077 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" event={"ID":"a07f22ff-a809-4060-a9d8-74944518d18e","Type":"ContainerDied","Data":"c892bb43e05c97559ed585af3ab195a868a8b308fb3ac5424cf5c75838c45aae"} Nov 03 20:22:03.334391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.334311 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:03.334682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.334650 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:03.360543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.348122 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x\" (UniqueName: \"kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x\") pod \"a07f22ff-a809-4060-a9d8-74944518d18e\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " Nov 03 20:22:03.513547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.476796 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume\") pod \"a07f22ff-a809-4060-a9d8-74944518d18e\" (UID: \"a07f22ff-a809-4060-a9d8-74944518d18e\") " Nov 03 20:22:03.513547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.478288 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume" pod "a07f22ff-a809-4060-a9d8-74944518d18e" (UID: "a07f22ff-a809-4060-a9d8-74944518d18e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:03.521549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.519237 5815 scope.go:122] "RemoveContainer" containerID="baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356" Nov 03 20:22:03.521549 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:03.520113 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356\": container with ID starting with baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356 not found: ID does not exist" containerID="baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356" Nov 03 20:22:03.521549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.520149 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356"} err="failed to get container status \"baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356\": rpc error: code = NotFound desc = could not find container \"baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356\": container with ID starting with baace39e2a62d3efc40ac44d30ca07bdfc8e64e2d67f40b76bca766758528356 not found: ID does not exist" Nov 03 20:22:03.521549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.520175 5815 scope.go:122] "RemoveContainer" containerID="cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b" Nov 03 20:22:03.549592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.549559 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq"] Nov 03 20:22:03.592338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.592243 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a07f22ff-a809-4060-a9d8-74944518d18e-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.636888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.636858 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-859b56f486-zs2nq"] Nov 03 20:22:03.781534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.767752 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x" pod "a07f22ff-a809-4060-a9d8-74944518d18e" (UID: "a07f22ff-a809-4060-a9d8-74944518d18e"). InnerVolumeSpecName "kube-api-access-2wm5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:03.843580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.835952 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wm5x\" (UniqueName: \"kubernetes.io/projected/a07f22ff-a809-4060-a9d8-74944518d18e-kube-api-access-2wm5x\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:03.853836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.846035 5815 scope.go:122] "RemoveContainer" containerID="cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b" Nov 03 20:22:03.872597 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:03.866887 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b\": container with ID starting with cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b not found: ID does not exist" containerID="cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b" Nov 03 20:22:03.872597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.866939 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b"} err="failed to get container status \"cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b\": rpc error: code = NotFound desc = could not find container \"cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b\": container with ID starting with cbe172a8a995c4c318e97c5554c81c4a5a6a61f26b9044563ddae7b03c303c5b not found: ID does not exist" Nov 03 20:22:03.996558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.996491 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:22:03.996950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:03.996794 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 20:22:04.002363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.001829 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz"] Nov 03 20:22:04.057145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.057115 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh"] Nov 03 20:22:04.101580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.097785 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:04.101580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.097841 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:04.346532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.344483 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:04.346532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.344582 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:04.415588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.393056 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9" Nov 03 20:22:04.505539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.499577 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9"] Nov 03 20:22:04.522533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.519687 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-659cb7bb8c-lgxr9"] Nov 03 20:22:04.700561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.694922 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c797e9f-208a-4378-97c8-78a93156f86a" path="/var/lib/kubelet/pods/5c797e9f-208a-4378-97c8-78a93156f86a/volumes" Nov 03 20:22:04.700561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.696137 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07f22ff-a809-4060-a9d8-74944518d18e" path="/var/lib/kubelet/pods/a07f22ff-a809-4060-a9d8-74944518d18e/volumes" Nov 03 20:22:04.700561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:04.696608 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:22:05.038813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:05.038780 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk"] Nov 03 20:22:05.041777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:05.041754 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b221c5e-4kbbk"] Nov 03 20:22:05.092532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:05.091548 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:05.092532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:05.091605 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:05.330961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:05.330862 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:05.331229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:05.331196 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:06.093897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:06.093861 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 20:22:06.094536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:06.094485 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 20:22:06.329625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:06.329584 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 20:22:06.329804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:06.329654 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 20:22:06.592198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:06.592160 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3674f5f6-5741-4844-b774-365b8943e79c" path="/var/lib/kubelet/pods/3674f5f6-5741-4844-b774-365b8943e79c/volumes" Nov 03 20:22:07.093715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.093687 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:22:07.094181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.094156 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:22:07.094990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.094971 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:22:07.333829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.333802 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:22:07.334651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.334630 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:22:07.830017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.829978 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" podUID="74ffaecf-165c-4ca9-9a70-8f416f8e0997" containerName="payload-processing" containerID="cri-o://7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76" gracePeriod=25 Nov 03 20:22:07.872825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:07.872785 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" podUID="d24ef639-e473-4fad-9e56-af609646954b" containerName="payload-pre-processing" containerID="cri-o://a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f" gracePeriod=25 Nov 03 20:22:08.221604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.221265 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:22:08.265632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.265611 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:22:08.305244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.305206 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8\" (UniqueName: \"kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8\") pod \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " Nov 03 20:22:08.305421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.305321 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv\" (UniqueName: \"kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv\") pod \"d24ef639-e473-4fad-9e56-af609646954b\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " Nov 03 20:22:08.305538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.305494 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume\") pod \"d24ef639-e473-4fad-9e56-af609646954b\" (UID: \"d24ef639-e473-4fad-9e56-af609646954b\") " Nov 03 20:22:08.307120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.307097 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume\") pod \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\" (UID: \"74ffaecf-165c-4ca9-9a70-8f416f8e0997\") " Nov 03 20:22:08.309295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.309262 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume" pod "d24ef639-e473-4fad-9e56-af609646954b" (UID: "d24ef639-e473-4fad-9e56-af609646954b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:08.309583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.309288 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume" pod "74ffaecf-165c-4ca9-9a70-8f416f8e0997" (UID: "74ffaecf-165c-4ca9-9a70-8f416f8e0997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:08.310544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.310438 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv" pod "d24ef639-e473-4fad-9e56-af609646954b" (UID: "d24ef639-e473-4fad-9e56-af609646954b"). InnerVolumeSpecName "kube-api-access-zcfwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:08.313972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.313951 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8" pod "74ffaecf-165c-4ca9-9a70-8f416f8e0997" (UID: "74ffaecf-165c-4ca9-9a70-8f416f8e0997"). InnerVolumeSpecName "kube-api-access-t7vp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:08.410340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.410244 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ffaecf-165c-4ca9-9a70-8f416f8e0997-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:08.410340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.410283 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vp8\" (UniqueName: \"kubernetes.io/projected/74ffaecf-165c-4ca9-9a70-8f416f8e0997-kube-api-access-t7vp8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:08.410340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.410298 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfwv\" (UniqueName: \"kubernetes.io/projected/d24ef639-e473-4fad-9e56-af609646954b-kube-api-access-zcfwv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:08.410340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.410311 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d24ef639-e473-4fad-9e56-af609646954b-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:08.586219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.586166 5815 generic.go:356] "Generic (PLEG): container finished" podID="74ffaecf-165c-4ca9-9a70-8f416f8e0997" containerID="7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76" exitCode=0 Nov 03 20:22:08.586752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.586724 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" Nov 03 20:22:08.588993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.588958 5815 generic.go:356] "Generic (PLEG): container finished" podID="d24ef639-e473-4fad-9e56-af609646954b" containerID="a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f" exitCode=0 Nov 03 20:22:08.589225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.589209 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" Nov 03 20:22:08.640538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.634027 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" event={"ID":"74ffaecf-165c-4ca9-9a70-8f416f8e0997","Type":"ContainerDied","Data":"7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76"} Nov 03 20:22:08.640538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.634095 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks" event={"ID":"74ffaecf-165c-4ca9-9a70-8f416f8e0997","Type":"ContainerDied","Data":"62f6bf52376e03b9685d7d38e0702bc3d222d2f9aadcf00e3826510f34581ac7"} Nov 03 20:22:08.640538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.634111 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" event={"ID":"d24ef639-e473-4fad-9e56-af609646954b","Type":"ContainerDied","Data":"a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f"} Nov 03 20:22:08.640538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.634127 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456" event={"ID":"d24ef639-e473-4fad-9e56-af609646954b","Type":"ContainerDied","Data":"f54208fa20411c51eb79aebca4c14acabba2920ff702a55deea25eada1bdb30e"} Nov 03 20:22:08.640538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.634151 5815 scope.go:122] "RemoveContainer" containerID="7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76" Nov 03 20:22:08.665622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.664752 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks"] Nov 03 20:22:08.737047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.736924 5815 scope.go:122] "RemoveContainer" containerID="7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76" Nov 03 20:22:08.737999 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:08.737488 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76\": container with ID starting with 7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76 not found: ID does not exist" containerID="7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76" Nov 03 20:22:08.737999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.737868 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76"} err="failed to get container status \"7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76\": rpc error: code = NotFound desc = could not find container \"7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76\": container with ID starting with 7857cf143860bd1f8ef2f66c51b3414c1f9c50c874e3d9ca0639954779a79c76 not found: ID does not exist" Nov 03 20:22:08.737999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.737898 5815 scope.go:122] "RemoveContainer" containerID="a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f" Nov 03 20:22:08.770219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.770111 5815 scope.go:122] "RemoveContainer" containerID="a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f" Nov 03 20:22:08.771085 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:08.770985 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f\": container with ID starting with a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f not found: ID does not exist" containerID="a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f" Nov 03 20:22:08.771085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.771021 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f"} err="failed to get container status \"a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f\": rpc error: code = NotFound desc = could not find container \"a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f\": container with ID starting with a10223e94d9bb974f55c5bcd96d48968187cef87d2539e2a1be9e5f2731d039f not found: ID does not exist" Nov 03 20:22:08.781913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.781836 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5bb486f988-vrdks"] Nov 03 20:22:08.788111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.788052 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456"] Nov 03 20:22:08.794912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:08.794878 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-5b5bcf6d7f-kt456"] Nov 03 20:22:09.002255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.002213 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" podUID="51f7797d-f982-4357-a209-b26e04c9cd31" containerName="payload-processing" containerID="cri-o://36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd" gracePeriod=25 Nov 03 20:22:09.059019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.058987 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" podUID="1b5d4152-feea-4abe-9c91-e02cf17f544b" containerName="payload-pre-processing" containerID="cri-o://960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e" gracePeriod=25 Nov 03 20:22:09.371876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.371852 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:22:09.373914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.373893 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:22:09.501341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.500485 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume\") pod \"51f7797d-f982-4357-a209-b26e04c9cd31\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " Nov 03 20:22:09.501341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.500553 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj\" (UniqueName: \"kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj\") pod \"51f7797d-f982-4357-a209-b26e04c9cd31\" (UID: \"51f7797d-f982-4357-a209-b26e04c9cd31\") " Nov 03 20:22:09.511686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.511640 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume" pod "51f7797d-f982-4357-a209-b26e04c9cd31" (UID: "51f7797d-f982-4357-a209-b26e04c9cd31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:09.517858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.512411 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume\") pod \"1b5d4152-feea-4abe-9c91-e02cf17f544b\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " Nov 03 20:22:09.517858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.512596 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp\" (UniqueName: \"kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp\") pod \"1b5d4152-feea-4abe-9c91-e02cf17f544b\" (UID: \"1b5d4152-feea-4abe-9c91-e02cf17f544b\") " Nov 03 20:22:09.517858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.513906 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume" pod "1b5d4152-feea-4abe-9c91-e02cf17f544b" (UID: "1b5d4152-feea-4abe-9c91-e02cf17f544b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:09.517858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.514626 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b5d4152-feea-4abe-9c91-e02cf17f544b-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:09.517858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.514652 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f7797d-f982-4357-a209-b26e04c9cd31-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:09.522811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.522650 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp" pod "1b5d4152-feea-4abe-9c91-e02cf17f544b" (UID: "1b5d4152-feea-4abe-9c91-e02cf17f544b"). InnerVolumeSpecName "kube-api-access-xj2fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:09.522811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.522778 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj" pod "51f7797d-f982-4357-a209-b26e04c9cd31" (UID: "51f7797d-f982-4357-a209-b26e04c9cd31"). InnerVolumeSpecName "kube-api-access-smxlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:09.605847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.605816 5815 generic.go:356] "Generic (PLEG): container finished" podID="1b5d4152-feea-4abe-9c91-e02cf17f544b" containerID="960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e" exitCode=0 Nov 03 20:22:09.606047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.605864 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" Nov 03 20:22:09.606047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.605855 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" event={"ID":"1b5d4152-feea-4abe-9c91-e02cf17f544b","Type":"ContainerDied","Data":"960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e"} Nov 03 20:22:09.606047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.606002 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh" event={"ID":"1b5d4152-feea-4abe-9c91-e02cf17f544b","Type":"ContainerDied","Data":"79f976cb3b92233d89eb1006ea24e7e5f78980094f8ef00882e8fd24505744df"} Nov 03 20:22:09.606047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.606034 5815 scope.go:122] "RemoveContainer" containerID="960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e" Nov 03 20:22:09.609010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.608982 5815 generic.go:356] "Generic (PLEG): container finished" podID="51f7797d-f982-4357-a209-b26e04c9cd31" containerID="36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd" exitCode=0 Nov 03 20:22:09.609119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.609078 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" event={"ID":"51f7797d-f982-4357-a209-b26e04c9cd31","Type":"ContainerDied","Data":"36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd"} Nov 03 20:22:09.609211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.609118 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" event={"ID":"51f7797d-f982-4357-a209-b26e04c9cd31","Type":"ContainerDied","Data":"e0089100c77a119d4270178bcea0c7dda631424b0f779cf6df881d0799bc650b"} Nov 03 20:22:09.609211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.609187 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz" Nov 03 20:22:09.615816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.615790 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxlj\" (UniqueName: \"kubernetes.io/projected/51f7797d-f982-4357-a209-b26e04c9cd31-kube-api-access-smxlj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:09.615937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.615819 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2fp\" (UniqueName: \"kubernetes.io/projected/1b5d4152-feea-4abe-9c91-e02cf17f544b-kube-api-access-xj2fp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:09.620017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.619848 5815 scope.go:122] "RemoveContainer" containerID="960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e" Nov 03 20:22:09.620224 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:09.620182 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e\": container with ID starting with 960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e not found: ID does not exist" containerID="960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e" Nov 03 20:22:09.620311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.620240 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e"} err="failed to get container status \"960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e\": rpc error: code = NotFound desc = could not find container \"960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e\": container with ID starting with 960151cbc35b79e3db1acac7a4585bf80a6642753cfea7edddd292dd5a61d03e not found: ID does not exist" Nov 03 20:22:09.620311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.620266 5815 scope.go:122] "RemoveContainer" containerID="36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd" Nov 03 20:22:09.630628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.630609 5815 scope.go:122] "RemoveContainer" containerID="36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd" Nov 03 20:22:09.630891 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:09.630874 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd\": container with ID starting with 36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd not found: ID does not exist" containerID="36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd" Nov 03 20:22:09.630955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.630898 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd"} err="failed to get container status \"36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd\": rpc error: code = NotFound desc = could not find container \"36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd\": container with ID starting with 36a22864301aa3d67c29c2989e06ba84b858fc97d38ca5248c06b3fcb573e8bd not found: ID does not exist" Nov 03 20:22:09.634080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.634056 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh"] Nov 03 20:22:09.640985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.640960 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-400777-7db698cc79-nshlh"] Nov 03 20:22:09.651873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.651851 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz"] Nov 03 20:22:09.657069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:09.657053 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-400777-776c774778-4mwtz"] Nov 03 20:22:10.245980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:10.245946 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8"] Nov 03 20:22:10.246273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:10.246220 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" containerID="cri-o://c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470" gracePeriod=30 Nov 03 20:22:10.595977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:10.591202 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b5d4152-feea-4abe-9c91-e02cf17f544b" path="/var/lib/kubelet/pods/1b5d4152-feea-4abe-9c91-e02cf17f544b/volumes" Nov 03 20:22:10.595977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:10.591815 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f7797d-f982-4357-a209-b26e04c9cd31" path="/var/lib/kubelet/pods/51f7797d-f982-4357-a209-b26e04c9cd31/volumes" Nov 03 20:22:10.595977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:10.592292 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ffaecf-165c-4ca9-9a70-8f416f8e0997" path="/var/lib/kubelet/pods/74ffaecf-165c-4ca9-9a70-8f416f8e0997/volumes" Nov 03 20:22:10.595977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:10.592827 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24ef639-e473-4fad-9e56-af609646954b" path="/var/lib/kubelet/pods/d24ef639-e473-4fad-9e56-af609646954b/volumes" Nov 03 20:22:13.448208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.448164 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd"] Nov 03 20:22:13.449165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449144 5815 cpu_manager.go:414] "containerMap: removing container" podUID="51f7797d-f982-4357-a209-b26e04c9cd31" containerName="payload-processing" Nov 03 20:22:13.449229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449168 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f7797d-f982-4357-a209-b26e04c9cd31" containerName="payload-processing" Nov 03 20:22:13.449229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449187 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1b5d4152-feea-4abe-9c91-e02cf17f544b" containerName="payload-pre-processing" Nov 03 20:22:13.449229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449194 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b5d4152-feea-4abe-9c91-e02cf17f544b" containerName="payload-pre-processing" Nov 03 20:22:13.449229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449221 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a07f22ff-a809-4060-a9d8-74944518d18e" containerName="payload-pre-processing" Nov 03 20:22:13.449229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449229 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07f22ff-a809-4060-a9d8-74944518d18e" containerName="payload-pre-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449276 5815 cpu_manager.go:414] "containerMap: removing container" podUID="74ffaecf-165c-4ca9-9a70-8f416f8e0997" containerName="payload-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449282 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ffaecf-165c-4ca9-9a70-8f416f8e0997" containerName="payload-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449302 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5c797e9f-208a-4378-97c8-78a93156f86a" containerName="payload-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449308 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c797e9f-208a-4378-97c8-78a93156f86a" containerName="payload-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449332 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d24ef639-e473-4fad-9e56-af609646954b" containerName="payload-pre-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449340 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24ef639-e473-4fad-9e56-af609646954b" containerName="payload-pre-processing" Nov 03 20:22:13.449486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449479 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ffaecf-165c-4ca9-9a70-8f416f8e0997" containerName="payload-processing" Nov 03 20:22:13.449878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449496 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c797e9f-208a-4378-97c8-78a93156f86a" containerName="payload-processing" Nov 03 20:22:13.449878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449534 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24ef639-e473-4fad-9e56-af609646954b" containerName="payload-pre-processing" Nov 03 20:22:13.449878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449547 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f7797d-f982-4357-a209-b26e04c9cd31" containerName="payload-processing" Nov 03 20:22:13.449878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449569 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b5d4152-feea-4abe-9c91-e02cf17f544b" containerName="payload-pre-processing" Nov 03 20:22:13.449878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.449581 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07f22ff-a809-4060-a9d8-74944518d18e" containerName="payload-pre-processing" Nov 03 20:22:13.455285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.455264 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.463914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.462985 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2eee8c\"" Nov 03 20:22:13.463914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.463084 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2eee8c-dockercfg-vjjtm\"" Nov 03 20:22:13.468104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.466754 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd"] Nov 03 20:22:13.624281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.607485 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.624281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.607575 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbdvx\" (UniqueName: \"kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.727106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.722547 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbdvx\" (UniqueName: \"kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.727106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.722819 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.734536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.731554 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.808564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:13.801275 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbdvx\" (UniqueName: \"kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx\") pod \"payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:13.860730 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:13.860690 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:22:14.073071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:14.072973 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:14.427052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:14.427026 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd"] Nov 03 20:22:14.427369 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:22:14.427348 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf95310_a3f1_4475_963e_a56d544d7ae3.slice/crio-c4b8470a717a55e3edbacef8ada5043dbeda44e795197fc84dd26fd1516b2f80 WatchSource:0}: Error finding container c4b8470a717a55e3edbacef8ada5043dbeda44e795197fc84dd26fd1516b2f80: Status 404 returned error can't find the container with id c4b8470a717a55e3edbacef8ada5043dbeda44e795197fc84dd26fd1516b2f80 Nov 03 20:22:14.695749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:14.695672 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" event={"ID":"9cf95310-a3f1-4475-963e-a56d544d7ae3","Type":"ContainerStarted","Data":"7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518"} Nov 03 20:22:14.695749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:14.695728 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" event={"ID":"9cf95310-a3f1-4475-963e-a56d544d7ae3","Type":"ContainerStarted","Data":"c4b8470a717a55e3edbacef8ada5043dbeda44e795197fc84dd26fd1516b2f80"} Nov 03 20:22:14.696298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:14.695757 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:14.718792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:14.718742 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" podStartSLOduration=1.7187285220000001 podStartE2EDuration="1.718728522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:14.716454548 +0000 UTC m=+1244.246800220" watchObservedRunningTime="2026-11-03 20:22:14.718728522 +0000 UTC m=+1244.249074194" Nov 03 20:22:15.548089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.548062 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.573684 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.573781 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.573944 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.573989 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.574038 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.574084 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.574140 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.574174 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.574280 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs\") pod \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\" (UID: \"1c676a88-ab43-48e5-ab69-8849f20ca3f7\") " Nov 03 20:22:15.575877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.575763 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.577107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.577087 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.579335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.578033 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.579335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.578415 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.579335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.579118 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:15.587919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.587890 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.588379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.588339 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:15.589831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.589797 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:15.590398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.590364 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb" pod "1c676a88-ab43-48e5-ab69-8849f20ca3f7" (UID: "1c676a88-ab43-48e5-ab69-8849f20ca3f7"). InnerVolumeSpecName "kube-api-access-5t8qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:15.676675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676650 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676674 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676683 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676691 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676700 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676709 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676716 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676725 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c676a88-ab43-48e5-ab69-8849f20ca3f7-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.676840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.676733 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8qb\" (UniqueName: \"kubernetes.io/projected/1c676a88-ab43-48e5-ab69-8849f20ca3f7-kube-api-access-5t8qb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.707175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.707139 5815 generic.go:356] "Generic (PLEG): container finished" podID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerID="c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470" exitCode=0 Nov 03 20:22:15.707726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.707206 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" Nov 03 20:22:15.707726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.707213 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" event={"ID":"1c676a88-ab43-48e5-ab69-8849f20ca3f7","Type":"ContainerDied","Data":"c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470"} Nov 03 20:22:15.707726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.707271 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8" event={"ID":"1c676a88-ab43-48e5-ab69-8849f20ca3f7","Type":"ContainerDied","Data":"f5bdab8c206f1ebdf863b952c0908d4452f2a69704388b3b868b1f913b389b4d"} Nov 03 20:22:15.707726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.707293 5815 scope.go:122] "RemoveContainer" containerID="c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470" Nov 03 20:22:15.719793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.719779 5815 scope.go:122] "RemoveContainer" containerID="c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470" Nov 03 20:22:15.720073 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:15.720057 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470\": container with ID starting with c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470 not found: ID does not exist" containerID="c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470" Nov 03 20:22:15.720143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.720086 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470"} err="failed to get container status \"c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470\": rpc error: code = NotFound desc = could not find container \"c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470\": container with ID starting with c8de28831147f1323dd54d23f0c8f977c3adaf18efca978fb35066ddf439c470 not found: ID does not exist" Nov 03 20:22:15.735349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.735323 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8"] Nov 03 20:22:15.739783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:15.739761 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-m9xd8"] Nov 03 20:22:16.024875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:16.024838 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4"] Nov 03 20:22:16.028925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:16.028902 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289c1d61-2x2x4"] Nov 03 20:22:16.588380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:16.588350 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" path="/var/lib/kubelet/pods/1c676a88-ab43-48e5-ab69-8849f20ca3f7/volumes" Nov 03 20:22:16.588902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:16.588886 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64dfbd0a-9a5a-4cd0-90ed-d88be7769e89" path="/var/lib/kubelet/pods/64dfbd0a-9a5a-4cd0-90ed-d88be7769e89/volumes" Nov 03 20:22:19.857010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:19.856777 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2"] Nov 03 20:22:19.857464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:19.857004 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" containerID="cri-o://2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a" gracePeriod=30 Nov 03 20:22:22.371538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:22.368691 5815 patch_prober.go:29] interesting pod/e2e-api-b-400777-openshift-default-845498b657-kg7b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:22:22.371538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:22.368773 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:22:23.155699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.155671 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9"] Nov 03 20:22:23.156130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.156079 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" podUID="d70ed4b2-8a73-432d-b12e-7c9e1a85c966" containerName="payload-processing" containerID="cri-o://cf790de80d04d9a61ed9bc830103e68e04294f930b5fe037b6e22ee2193cd121" gracePeriod=30 Nov 03 20:22:23.226821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.226792 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs"] Nov 03 20:22:23.227356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.227323 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" podUID="d38a226e-b77f-4821-8f2c-87681e3830e0" containerName="payload-pre-processing" containerID="cri-o://92d686becb2268929e5a961535a8092883d7b3781ed7fb9564273e5bfbf9f73a" gracePeriod=30 Nov 03 20:22:23.826018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.817118 5815 generic.go:356] "Generic (PLEG): container finished" podID="d70ed4b2-8a73-432d-b12e-7c9e1a85c966" containerID="cf790de80d04d9a61ed9bc830103e68e04294f930b5fe037b6e22ee2193cd121" exitCode=0 Nov 03 20:22:23.826018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.817402 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" event={"ID":"d70ed4b2-8a73-432d-b12e-7c9e1a85c966","Type":"ContainerDied","Data":"cf790de80d04d9a61ed9bc830103e68e04294f930b5fe037b6e22ee2193cd121"} Nov 03 20:22:23.826018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.819998 5815 generic.go:356] "Generic (PLEG): container finished" podID="d38a226e-b77f-4821-8f2c-87681e3830e0" containerID="92d686becb2268929e5a961535a8092883d7b3781ed7fb9564273e5bfbf9f73a" exitCode=0 Nov 03 20:22:23.826018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:23.820177 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" event={"ID":"d38a226e-b77f-4821-8f2c-87681e3830e0","Type":"ContainerDied","Data":"92d686becb2268929e5a961535a8092883d7b3781ed7fb9564273e5bfbf9f73a"} Nov 03 20:22:24.062054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.062029 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:22:24.145249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.145158 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw\" (UniqueName: \"kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw\") pod \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " Nov 03 20:22:24.149313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.149281 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw" pod "d70ed4b2-8a73-432d-b12e-7c9e1a85c966" (UID: "d70ed4b2-8a73-432d-b12e-7c9e1a85c966"). InnerVolumeSpecName "kube-api-access-vtlkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:24.178525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.174818 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.247183 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp\" (UniqueName: \"kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp\") pod \"d38a226e-b77f-4821-8f2c-87681e3830e0\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.247254 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume\") pod \"d38a226e-b77f-4821-8f2c-87681e3830e0\" (UID: \"d38a226e-b77f-4821-8f2c-87681e3830e0\") " Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.247376 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume\") pod \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\" (UID: \"d70ed4b2-8a73-432d-b12e-7c9e1a85c966\") " Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.247972 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlkw\" (UniqueName: \"kubernetes.io/projected/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-kube-api-access-vtlkw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.248357 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume" pod "d70ed4b2-8a73-432d-b12e-7c9e1a85c966" (UID: "d70ed4b2-8a73-432d-b12e-7c9e1a85c966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.248980 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume" pod "d38a226e-b77f-4821-8f2c-87681e3830e0" (UID: "d38a226e-b77f-4821-8f2c-87681e3830e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:24.252558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.250702 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp" pod "d38a226e-b77f-4821-8f2c-87681e3830e0" (UID: "d38a226e-b77f-4821-8f2c-87681e3830e0"). InnerVolumeSpecName "kube-api-access-c7xtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:24.349606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.349575 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xtp\" (UniqueName: \"kubernetes.io/projected/d38a226e-b77f-4821-8f2c-87681e3830e0-kube-api-access-c7xtp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:24.349606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.349603 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38a226e-b77f-4821-8f2c-87681e3830e0-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:24.349606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.349612 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ed4b2-8a73-432d-b12e-7c9e1a85c966-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:24.832969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.832936 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" Nov 03 20:22:24.832969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.832959 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs" event={"ID":"d38a226e-b77f-4821-8f2c-87681e3830e0","Type":"ContainerDied","Data":"4c7cd0ca1e561c7b49395924c3fc173e0eedf99f842f95432ca1e1f4aa38e31b"} Nov 03 20:22:24.833473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.833024 5815 scope.go:122] "RemoveContainer" containerID="92d686becb2268929e5a961535a8092883d7b3781ed7fb9564273e5bfbf9f73a" Nov 03 20:22:24.834991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.834967 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" Nov 03 20:22:24.835163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.835013 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9" event={"ID":"d70ed4b2-8a73-432d-b12e-7c9e1a85c966","Type":"ContainerDied","Data":"e432346cb519e0024c11739e72927980b96b57ff96beefcdecaa679fb30e56eb"} Nov 03 20:22:24.845023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.844998 5815 scope.go:122] "RemoveContainer" containerID="cf790de80d04d9a61ed9bc830103e68e04294f930b5fe037b6e22ee2193cd121" Nov 03 20:22:24.858192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.858168 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs"] Nov 03 20:22:24.868081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.868056 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-6955f99bf9-gtfqs"] Nov 03 20:22:24.881563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.881443 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9"] Nov 03 20:22:24.887610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:24.887580 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-6f5c4b4944-c26g9"] Nov 03 20:22:25.153855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.153829 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:22:25.176123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176088 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.176311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176139 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.176311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176180 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.176538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176385 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:25.176682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176649 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.176772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176697 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.176772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176762 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.176969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.176872 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.177115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.177039 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.177209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.177117 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl\") pod \"c18d069c-790a-483d-a6bd-78ce0e56216a\" (UID: \"c18d069c-790a-483d-a6bd-78ce0e56216a\") " Nov 03 20:22:25.177811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.177790 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.180420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.180276 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:25.180420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.180370 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:25.180650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.180628 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:25.180801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.180782 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:25.183442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.183418 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:25.184086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.184035 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:25.185876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.185810 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "kube-api-access-7dzzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:25.186926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.186902 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token" pod "c18d069c-790a-483d-a6bd-78ce0e56216a" (UID: "c18d069c-790a-483d-a6bd-78ce0e56216a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279194 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279217 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279226 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279237 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279247 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279255 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18d069c-790a-483d-a6bd-78ce0e56216a-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279262 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18d069c-790a-483d-a6bd-78ce0e56216a-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.279765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.279294 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzzl\" (UniqueName: \"kubernetes.io/projected/c18d069c-790a-483d-a6bd-78ce0e56216a-kube-api-access-7dzzl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:25.709698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.709619 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:22:25.847302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.847255 5815 generic.go:356] "Generic (PLEG): container finished" podID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerID="2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a" exitCode=0 Nov 03 20:22:25.847302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.847287 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" event={"ID":"c18d069c-790a-483d-a6bd-78ce0e56216a","Type":"ContainerDied","Data":"2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a"} Nov 03 20:22:25.847864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.847319 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" Nov 03 20:22:25.847864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.847346 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2" event={"ID":"c18d069c-790a-483d-a6bd-78ce0e56216a","Type":"ContainerDied","Data":"c0d7d3cf0909938d29cb627ecbc4b5a188b8b6fd2fdd3f154909b7ef85c623fa"} Nov 03 20:22:25.847864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.847365 5815 scope.go:122] "RemoveContainer" containerID="2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a" Nov 03 20:22:25.863897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.863875 5815 scope.go:122] "RemoveContainer" containerID="17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58" Nov 03 20:22:25.875185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.875095 5815 scope.go:122] "RemoveContainer" containerID="2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a" Nov 03 20:22:25.875457 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:25.875421 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a\": container with ID starting with 2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a not found: ID does not exist" containerID="2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a" Nov 03 20:22:25.875555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.875460 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a"} err="failed to get container status \"2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a\": rpc error: code = NotFound desc = could not find container \"2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a\": container with ID starting with 2d6e1a50a6ef53cad16821bd08802a31ed756c132bc8eecc2df6b4bf3bc43a3a not found: ID does not exist" Nov 03 20:22:25.875555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.875483 5815 scope.go:122] "RemoveContainer" containerID="17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58" Nov 03 20:22:25.875687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.875677 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2"] Nov 03 20:22:25.875854 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:25.875830 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58\": container with ID starting with 17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58 not found: ID does not exist" containerID="17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58" Nov 03 20:22:25.875911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.875862 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58"} err="failed to get container status \"17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58\": rpc error: code = NotFound desc = could not find container \"17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58\": container with ID starting with 17137012dc7ef090da84b7a745fc5bf54991890968337582c4d42e8c25d1bb58 not found: ID does not exist" Nov 03 20:22:25.879412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:25.879389 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-400777-openshift-default-845498b657-kg7b2"] Nov 03 20:22:26.589725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:26.589694 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" path="/var/lib/kubelet/pods/c18d069c-790a-483d-a6bd-78ce0e56216a/volumes" Nov 03 20:22:26.590143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:26.590128 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38a226e-b77f-4821-8f2c-87681e3830e0" path="/var/lib/kubelet/pods/d38a226e-b77f-4821-8f2c-87681e3830e0/volumes" Nov 03 20:22:26.590473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:26.590461 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70ed4b2-8a73-432d-b12e-7c9e1a85c966" path="/var/lib/kubelet/pods/d70ed4b2-8a73-432d-b12e-7c9e1a85c966/volumes" Nov 03 20:22:26.883219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:26.883136 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:22:27.733899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.733861 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9"] Nov 03 20:22:27.734744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734724 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d70ed4b2-8a73-432d-b12e-7c9e1a85c966" containerName="payload-processing" Nov 03 20:22:27.734833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734745 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70ed4b2-8a73-432d-b12e-7c9e1a85c966" containerName="payload-processing" Nov 03 20:22:27.734833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734781 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" Nov 03 20:22:27.734833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734789 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" Nov 03 20:22:27.734833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734813 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" Nov 03 20:22:27.734833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734821 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734841 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734848 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734861 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d38a226e-b77f-4821-8f2c-87681e3830e0" containerName="payload-pre-processing" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734867 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38a226e-b77f-4821-8f2c-87681e3830e0" containerName="payload-pre-processing" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.734996 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.735009 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18d069c-790a-483d-a6bd-78ce0e56216a" containerName="istio-proxy" Nov 03 20:22:27.735027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.735023 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c676a88-ab43-48e5-ab69-8849f20ca3f7" containerName="istio-proxy" Nov 03 20:22:27.735254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.735034 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38a226e-b77f-4821-8f2c-87681e3830e0" containerName="payload-pre-processing" Nov 03 20:22:27.735254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.735050 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70ed4b2-8a73-432d-b12e-7c9e1a85c966" containerName="payload-processing" Nov 03 20:22:27.740575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.740555 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:27.747158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.746036 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-82c116-dockercfg-c9b2g\"" Nov 03 20:22:27.747158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.746113 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-82c116\"" Nov 03 20:22:27.749840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.749813 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9"] Nov 03 20:22:27.760483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.760452 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z"] Nov 03 20:22:27.766826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.766805 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:27.779664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.779638 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z"] Nov 03 20:22:27.818233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.818206 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzmr\" (UniqueName: \"kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr\") pod \"payload-processing-e2e-praxis-a-82c116-747858446-4c54z\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:27.818400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.818244 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:27.818573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.818557 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmn8d\" (UniqueName: \"kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d\") pod \"payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:27.818643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.818613 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-82c116-747858446-4c54z\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:27.924531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.922200 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmn8d\" (UniqueName: \"kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d\") pod \"payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:27.924531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.922400 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-82c116-747858446-4c54z\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:27.924531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.922534 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzmr\" (UniqueName: \"kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr\") pod \"payload-processing-e2e-praxis-a-82c116-747858446-4c54z\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:27.924531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.922559 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:27.937544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.936595 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-82c116-747858446-4c54z\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:27.937745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:27.937696 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:28.056654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:28.055891 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmn8d\" (UniqueName: \"kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d\") pod \"payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:28.066418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:28.066373 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzmr\" (UniqueName: \"kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr\") pod \"payload-processing-e2e-praxis-a-82c116-747858446-4c54z\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:28.097456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:28.096790 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:28.097456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:28.097315 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:28.760422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:28.760391 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9"] Nov 03 20:22:28.760854 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:22:28.760822 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67aace8_4a04_47d8_a473_d99b7fa8531e.slice/crio-67d383542069e14398fcb86b3a5e40d1c86f2fca028d5df06036ac84707cdeac WatchSource:0}: Error finding container 67d383542069e14398fcb86b3a5e40d1c86f2fca028d5df06036ac84707cdeac: Status 404 returned error can't find the container with id 67d383542069e14398fcb86b3a5e40d1c86f2fca028d5df06036ac84707cdeac Nov 03 20:22:28.873819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:28.873786 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z"] Nov 03 20:22:28.874095 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:22:28.874070 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f6baa9_816c_4229_9788_008da7b9ccd2.slice/crio-98f488e21c892fa0943cb933453b44a18feb2942f028adad60f64f380ac16e4c WatchSource:0}: Error finding container 98f488e21c892fa0943cb933453b44a18feb2942f028adad60f64f380ac16e4c: Status 404 returned error can't find the container with id 98f488e21c892fa0943cb933453b44a18feb2942f028adad60f64f380ac16e4c Nov 03 20:22:29.083602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.081631 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" event={"ID":"f6f6baa9-816c-4229-9788-008da7b9ccd2","Type":"ContainerStarted","Data":"f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1"} Nov 03 20:22:29.083602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.081691 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" event={"ID":"f6f6baa9-816c-4229-9788-008da7b9ccd2","Type":"ContainerStarted","Data":"98f488e21c892fa0943cb933453b44a18feb2942f028adad60f64f380ac16e4c"} Nov 03 20:22:29.083602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.081743 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:29.084193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.083885 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" event={"ID":"d67aace8-4a04-47d8-a473-d99b7fa8531e","Type":"ContainerStarted","Data":"95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76"} Nov 03 20:22:29.084193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.083925 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" event={"ID":"d67aace8-4a04-47d8-a473-d99b7fa8531e","Type":"ContainerStarted","Data":"67d383542069e14398fcb86b3a5e40d1c86f2fca028d5df06036ac84707cdeac"} Nov 03 20:22:29.084193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.083950 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:29.104440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.104387 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" podStartSLOduration=2.104374912 podStartE2EDuration="2.104374912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:29.099872128 +0000 UTC m=+1258.630217802" watchObservedRunningTime="2026-11-03 20:22:29.104374912 +0000 UTC m=+1258.634720583" Nov 03 20:22:29.121420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.121374 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" podStartSLOduration=2.121361964 podStartE2EDuration="2.121361964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:29.117992069 +0000 UTC m=+1258.648337752" watchObservedRunningTime="2026-11-03 20:22:29.121361964 +0000 UTC m=+1258.651707642" Nov 03 20:22:29.643415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.643380 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8"] Nov 03 20:22:29.643682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:29.643652 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" containerID="cri-o://8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743" gracePeriod=30 Nov 03 20:22:30.171337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.171300 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm"] Nov 03 20:22:30.190229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.185280 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.240110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.191902 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1142fd-openshift-default-dockercfg-znktn\"" Nov 03 20:22:30.240110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.230583 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm"] Nov 03 20:22:30.253391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.252969 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.253391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.253116 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.256310 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.256401 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.256498 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.256624 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rs6w\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.256770 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.256898 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.259345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.257017 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.353634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.350277 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr"] Nov 03 20:22:30.353634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.350548 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" podUID="ffba69fd-250c-4f60-bbc3-6278de6bace0" containerName="payload-pre-processing" containerID="cri-o://88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1" gracePeriod=30 Nov 03 20:22:30.361089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.360555 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.362044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.362024 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.362218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.362203 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.362377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.362361 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.362699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.362679 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.362845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.362830 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.362950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.362936 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.363074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.363061 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rs6w\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.363251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.363236 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.366284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.364469 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.366284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.365143 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.366284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.365377 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.366284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.365641 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.366284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.366123 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.408754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.405789 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.409477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.409073 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.410758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.410736 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rs6w\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.429871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.429818 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token\") pod \"e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.598535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.597731 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:30.948410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:30.948384 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:22:31.063695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.062100 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs\" (UniqueName: \"kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs\") pod \"ffba69fd-250c-4f60-bbc3-6278de6bace0\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " Nov 03 20:22:31.063695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.062291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume\") pod \"ffba69fd-250c-4f60-bbc3-6278de6bace0\" (UID: \"ffba69fd-250c-4f60-bbc3-6278de6bace0\") " Nov 03 20:22:31.064189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.064146 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume" pod "ffba69fd-250c-4f60-bbc3-6278de6bace0" (UID: "ffba69fd-250c-4f60-bbc3-6278de6bace0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:31.069846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.069739 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs" pod "ffba69fd-250c-4f60-bbc3-6278de6bace0" (UID: "ffba69fd-250c-4f60-bbc3-6278de6bace0"). InnerVolumeSpecName "kube-api-access-6h5rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:31.106166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.106133 5815 generic.go:356] "Generic (PLEG): container finished" podID="ffba69fd-250c-4f60-bbc3-6278de6bace0" containerID="88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1" exitCode=0 Nov 03 20:22:31.106386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.106366 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" event={"ID":"ffba69fd-250c-4f60-bbc3-6278de6bace0","Type":"ContainerDied","Data":"88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1"} Nov 03 20:22:31.106463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.106426 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" event={"ID":"ffba69fd-250c-4f60-bbc3-6278de6bace0","Type":"ContainerDied","Data":"b1d6960aaa954c617606e3203a57ebcef399a07f8790927cda88c1c1980fcb7d"} Nov 03 20:22:31.106559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.106461 5815 scope.go:122] "RemoveContainer" containerID="88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1" Nov 03 20:22:31.106693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.106648 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr" Nov 03 20:22:31.123226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.123208 5815 scope.go:122] "RemoveContainer" containerID="88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1" Nov 03 20:22:31.123495 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:31.123477 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1\": container with ID starting with 88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1 not found: ID does not exist" containerID="88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1" Nov 03 20:22:31.123658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.123525 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1"} err="failed to get container status \"88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1\": rpc error: code = NotFound desc = could not find container \"88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1\": container with ID starting with 88a14b0ce1dda6c495c30b6be4d48dfe72e1370004bba77184eda522df1c0ad1 not found: ID does not exist" Nov 03 20:22:31.149858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.149820 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr"] Nov 03 20:22:31.155876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.155851 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-5bd49584dc-f5chr"] Nov 03 20:22:31.163939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.163917 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba69fd-250c-4f60-bbc3-6278de6bace0-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:31.164026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.163944 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5rs\" (UniqueName: \"kubernetes.io/projected/ffba69fd-250c-4f60-bbc3-6278de6bace0-kube-api-access-6h5rs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:31.223382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.223345 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm"] Nov 03 20:22:31.224601 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:22:31.224567 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b80b43d_1822_4c79_8a0f_bc60123fdf41.slice/crio-8b9c8a76f3bd0583e5d94ee6edb0f010f12acf5d6dd730c3d38d69c53454edac WatchSource:0}: Error finding container 8b9c8a76f3bd0583e5d94ee6edb0f010f12acf5d6dd730c3d38d69c53454edac: Status 404 returned error can't find the container with id 8b9c8a76f3bd0583e5d94ee6edb0f010f12acf5d6dd730c3d38d69c53454edac Nov 03 20:22:31.228132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.228091 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:22:31.228226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.228176 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:22:31.228282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:31.228222 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:22:32.118019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:32.117982 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" event={"ID":"9b80b43d-1822-4c79-8a0f-bc60123fdf41","Type":"ContainerStarted","Data":"03369573c818494f7d5d3c6e9d3094d9e1d5bcac8262f3581e5c8a6535864e7b"} Nov 03 20:22:32.118309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:32.118286 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" event={"ID":"9b80b43d-1822-4c79-8a0f-bc60123fdf41","Type":"ContainerStarted","Data":"8b9c8a76f3bd0583e5d94ee6edb0f010f12acf5d6dd730c3d38d69c53454edac"} Nov 03 20:22:32.142767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:32.142721 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podStartSLOduration=2.142707859 podStartE2EDuration="2.142707859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:32.140304873 +0000 UTC m=+1261.670650545" watchObservedRunningTime="2026-11-03 20:22:32.142707859 +0000 UTC m=+1261.673053531" Nov 03 20:22:32.671652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:32.671615 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffba69fd-250c-4f60-bbc3-6278de6bace0" path="/var/lib/kubelet/pods/ffba69fd-250c-4f60-bbc3-6278de6bace0/volumes" Nov 03 20:22:32.672229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:32.672069 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:32.672229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:32.672177 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:33.043897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.043861 5815 scope.go:122] "RemoveContainer" containerID="17390cb5e887b8695875affb632ff71a5885eb88b5e5035c03edfa10147b476d" Nov 03 20:22:33.055088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.055054 5815 scope.go:122] "RemoveContainer" containerID="48c5004cd47e1780d5d3f8d7b6c21355084827e9a3304c36109fc07d443d618d" Nov 03 20:22:33.065382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.065359 5815 scope.go:122] "RemoveContainer" containerID="8a88496479e0584397b52726a0f76ff60cb51ebf2a1ac255547cbe628780baf1" Nov 03 20:22:33.077625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.077602 5815 scope.go:122] "RemoveContainer" containerID="ba5b70171f804a3382f3593813f4e6f678d794ebee63d7cb14c37250cf701cba" Nov 03 20:22:33.089080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.089063 5815 scope.go:122] "RemoveContainer" containerID="92faaef97786a5cf1cdf055b71a1a944184ea7043533d569039cf922034f0294" Nov 03 20:22:33.100418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.100399 5815 scope.go:122] "RemoveContainer" containerID="a017279d7fcc3da36a2fe69f45a3c6975301889f3bd4e2c64cb937e5ebca077f" Nov 03 20:22:33.114334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.114316 5815 scope.go:122] "RemoveContainer" containerID="e4f67bfaaa8dc88f8cfee1b4e389ba6f4e77d4f626b79ae7f935c3aca98ad1a0" Nov 03 20:22:33.124634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.124614 5815 scope.go:122] "RemoveContainer" containerID="709e2d9fa36e3d858a5c7f96d31d25164cc4d8fa0d57cd5f3aca2e0b6878eef7" Nov 03 20:22:33.141883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.141862 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:33.142648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.142627 5815 scope.go:122] "RemoveContainer" containerID="70b1e7a90d6fc2a44496fcca7e608dcdc270725369f6453331c29fbe2998724c" Nov 03 20:22:33.142932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.142915 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:22:33.152802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.152783 5815 scope.go:122] "RemoveContainer" containerID="74a6def0f8b33941b229f7e82ae957a3bfb715793b7e8d8f78e36e9234e9e9ea" Nov 03 20:22:33.162165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.162145 5815 scope.go:122] "RemoveContainer" containerID="1cdc570b7f905d3f42bc4e63833cf6662e1aaab12fed0791f35239a9ebbe12e2" Nov 03 20:22:33.185373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:33.185351 5815 scope.go:122] "RemoveContainer" containerID="cbd909939f5b182738e6a2356d1d88ac4ba53755ebdd7f1113373ee788415961" Nov 03 20:22:35.035268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.035246 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:22:35.131330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131304 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131337 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131374 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131423 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131591 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131622 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131633 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:35.131796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131690 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131738 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131810 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo\") pod \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\" (UID: \"bfb070e8-cb4e-467b-b35d-5c8b0a77e475\") " Nov 03 20:22:35.131929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131843 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:35.131929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131891 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:35.131929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.131823 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:35.132137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.132077 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:35.132859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.132833 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.132977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.132862 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.132977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.132877 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.132977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.132891 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.132977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.132904 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.134115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.134086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:35.134313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.134291 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "kube-api-access-djvj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:35.134581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.134561 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:35.134716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.134696 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token" pod "bfb070e8-cb4e-467b-b35d-5c8b0a77e475" (UID: "bfb070e8-cb4e-467b-b35d-5c8b0a77e475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:35.170474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.170437 5815 generic.go:356] "Generic (PLEG): container finished" podID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerID="8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743" exitCode=0 Nov 03 20:22:35.171384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.171360 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" Nov 03 20:22:35.174218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.171709 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" event={"ID":"bfb070e8-cb4e-467b-b35d-5c8b0a77e475","Type":"ContainerDied","Data":"8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743"} Nov 03 20:22:35.174218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.171767 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8" event={"ID":"bfb070e8-cb4e-467b-b35d-5c8b0a77e475","Type":"ContainerDied","Data":"c1daa7f1d8e496aa805fe3639fabb893bcb03aee8606d0c1bf7652181ad7bcae"} Nov 03 20:22:35.174218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.171797 5815 scope.go:122] "RemoveContainer" containerID="8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743" Nov 03 20:22:35.188314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.188297 5815 scope.go:122] "RemoveContainer" containerID="09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162" Nov 03 20:22:35.203295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.203275 5815 scope.go:122] "RemoveContainer" containerID="8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743" Nov 03 20:22:35.203617 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:35.203595 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743\": container with ID starting with 8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743 not found: ID does not exist" containerID="8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743" Nov 03 20:22:35.203704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.203623 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743"} err="failed to get container status \"8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743\": rpc error: code = NotFound desc = could not find container \"8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743\": container with ID starting with 8e5f76156d341bd2963b2f6b42178cea3a39b42d0bfb16dd82f291907cab5743 not found: ID does not exist" Nov 03 20:22:35.203704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.203643 5815 scope.go:122] "RemoveContainer" containerID="09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162" Nov 03 20:22:35.204216 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:35.204192 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162\": container with ID starting with 09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162 not found: ID does not exist" containerID="09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162" Nov 03 20:22:35.204302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.204224 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162"} err="failed to get container status \"09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162\": rpc error: code = NotFound desc = could not find container \"09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162\": container with ID starting with 09cdf95bec0800fe98cc02ace3f9a79d9427f49b4933b44689e601c98aa73162 not found: ID does not exist" Nov 03 20:22:35.211855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.211834 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8"] Nov 03 20:22:35.219701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.219677 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d6789f-openshift-default-7658fb4fc7-hx4d8"] Nov 03 20:22:35.234236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.234212 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvj4\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-kube-api-access-djvj4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.234236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.234236 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.234366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.234247 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.234366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.234255 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfb070e8-cb4e-467b-b35d-5c8b0a77e475-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:35.403296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.403212 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj"] Nov 03 20:22:35.404173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404154 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" Nov 03 20:22:35.404236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404178 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" Nov 03 20:22:35.404236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404206 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" Nov 03 20:22:35.404236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404215 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" Nov 03 20:22:35.404330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404250 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ffba69fd-250c-4f60-bbc3-6278de6bace0" containerName="payload-pre-processing" Nov 03 20:22:35.404330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404258 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffba69fd-250c-4f60-bbc3-6278de6bace0" containerName="payload-pre-processing" Nov 03 20:22:35.404428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404416 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffba69fd-250c-4f60-bbc3-6278de6bace0" containerName="payload-pre-processing" Nov 03 20:22:35.404460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404443 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" Nov 03 20:22:35.404460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.404456 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" containerName="istio-proxy" Nov 03 20:22:35.409997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.409977 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.414569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.414498 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efba68\"" Nov 03 20:22:35.417885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.417864 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj"] Nov 03 20:22:35.558630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.552950 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efba68-dockercfg-jc69g\"" Nov 03 20:22:35.558630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.556251 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww4j\" (UniqueName: \"kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j\") pod \"payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.558630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.556325 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.620530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.619612 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h"] Nov 03 20:22:35.633530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.631884 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:35.675592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.657930 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww4j\" (UniqueName: \"kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j\") pod \"payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.675915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.675891 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.677026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.660453 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h"] Nov 03 20:22:35.677183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.677158 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.686386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.686328 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww4j\" (UniqueName: \"kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j\") pod \"payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.768713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.760084 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:35.822769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.821704 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wx6\" (UniqueName: \"kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6\") pod \"payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:35.822769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.821843 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:35.924345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.923329 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wx6\" (UniqueName: \"kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6\") pod \"payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:35.924345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.923415 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:35.924345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.924163 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:35.942328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:35.942270 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wx6\" (UniqueName: \"kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6\") pod \"payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:36.049073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:36.049032 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:36.254601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:36.246367 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj"] Nov 03 20:22:36.598122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:36.598087 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb070e8-cb4e-467b-b35d-5c8b0a77e475" path="/var/lib/kubelet/pods/bfb070e8-cb4e-467b-b35d-5c8b0a77e475/volumes" Nov 03 20:22:36.698329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:36.698304 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h"] Nov 03 20:22:36.701532 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:22:36.701459 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc307b8e8_e026_4aeb_837b_1e3dcb3c1cc4.slice/crio-8e97b79a85ba0dbd55831cd452bac1a3a637422c2f09d512ca71eb1ecfb6346f WatchSource:0}: Error finding container 8e97b79a85ba0dbd55831cd452bac1a3a637422c2f09d512ca71eb1ecfb6346f: Status 404 returned error can't find the container with id 8e97b79a85ba0dbd55831cd452bac1a3a637422c2f09d512ca71eb1ecfb6346f Nov 03 20:22:37.322908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.322873 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" event={"ID":"1dc5ceba-3c75-4191-a691-a578b0c4efe7","Type":"ContainerStarted","Data":"4e2660ff0b2d246b1249cd779ffc3743046609a8f7a436433a448fcfdf57cd8b"} Nov 03 20:22:37.323380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.322932 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:37.323380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.322954 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" event={"ID":"1dc5ceba-3c75-4191-a691-a578b0c4efe7","Type":"ContainerStarted","Data":"ce0db69a352b3824f6ef31f65b835a123309010c6d0a93731fae5c539f0c1e06"} Nov 03 20:22:37.324669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.324644 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" event={"ID":"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4","Type":"ContainerStarted","Data":"6494d20d521b8281d0f0a90962e6aac41e3e6d3357cdf162ea42ce92343fcb0b"} Nov 03 20:22:37.324783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.324690 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" event={"ID":"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4","Type":"ContainerStarted","Data":"8e97b79a85ba0dbd55831cd452bac1a3a637422c2f09d512ca71eb1ecfb6346f"} Nov 03 20:22:37.324783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.324723 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:37.344522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.344469 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" podStartSLOduration=2.344454975 podStartE2EDuration="2.344454975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:37.340826572 +0000 UTC m=+1266.871172243" watchObservedRunningTime="2026-11-03 20:22:37.344454975 +0000 UTC m=+1266.874800646" Nov 03 20:22:37.361171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:37.361132 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" podStartSLOduration=2.361119162 podStartE2EDuration="2.361119162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:37.359805553 +0000 UTC m=+1266.890151224" watchObservedRunningTime="2026-11-03 20:22:37.361119162 +0000 UTC m=+1266.891464873" Nov 03 20:22:40.096481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:40.096403 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:22:40.096957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:40.096583 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:22:42.045077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.042736 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg"] Nov 03 20:22:42.051178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.051142 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.063704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.063681 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a596f7-openshift-default-dockercfg-jbqkz\"" Nov 03 20:22:42.065824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.065797 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg"] Nov 03 20:22:42.260534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260424 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260486 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260554 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260583 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260679 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260711 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260744 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260779 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.260985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.260913 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rddg2\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363150 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363225 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363277 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363303 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363416 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363454 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363487 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363544 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.363717 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rddg2\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.365587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.364859 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.366236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.366015 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.366293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.366248 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.367173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.367150 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.367656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.367597 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.403140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.401195 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.405834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.405774 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.412556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.408236 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rddg2\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.413151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.413101 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token\") pod \"e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:42.671998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:42.671913 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:43.062644 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:22:43.062610 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094af966_0312_4dbe_9145_5ee078f696c8.slice/crio-e1d527e04983e25d05142f2b832b1071d675889b4a7b413d8a2b320b461247cb WatchSource:0}: Error finding container e1d527e04983e25d05142f2b832b1071d675889b4a7b413d8a2b320b461247cb: Status 404 returned error can't find the container with id e1d527e04983e25d05142f2b832b1071d675889b4a7b413d8a2b320b461247cb Nov 03 20:22:43.065056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.065016 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:22:43.065137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.065109 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:22:43.065185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.065152 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:22:43.067520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.067486 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg"] Nov 03 20:22:43.587574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.587481 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" event={"ID":"094af966-0312-4dbe-9145-5ee078f696c8","Type":"ContainerStarted","Data":"d3cb3849b62c34bc0bd23fc4755f03fa904fce2ee263a9fb15ead5b9807a9217"} Nov 03 20:22:43.587752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.587595 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" event={"ID":"094af966-0312-4dbe-9145-5ee078f696c8","Type":"ContainerStarted","Data":"e1d527e04983e25d05142f2b832b1071d675889b4a7b413d8a2b320b461247cb"} Nov 03 20:22:43.617387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.617324 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podStartSLOduration=1.617306502 podStartE2EDuration="1.617306502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:43.611087542 +0000 UTC m=+1273.141433227" watchObservedRunningTime="2026-11-03 20:22:43.617306502 +0000 UTC m=+1273.147652178" Nov 03 20:22:43.672836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:43.672810 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:43.860067 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:22:43.859970 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:22:44.679384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:44.679341 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:45.605562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:45.605164 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:45.606787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:45.606761 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:22:46.026231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:46.026200 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz"] Nov 03 20:22:46.035712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:46.035683 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6d296a9-dhckz"] Nov 03 20:22:46.590320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:46.590287 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="183fa9c4-eeb9-46d3-bea0-be98263488d3" path="/var/lib/kubelet/pods/183fa9c4-eeb9-46d3-bea0-be98263488d3/volumes" Nov 03 20:22:47.042724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:47.042688 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf"] Nov 03 20:22:47.045342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:47.045319 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b679c5a5-tftwf"] Nov 03 20:22:48.336713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:48.336684 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:22:48.337137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:48.336772 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:22:48.659990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:48.659898 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6f982a-4ea4-4b7e-a4c2-77b261796484" path="/var/lib/kubelet/pods/ac6f982a-4ea4-4b7e-a4c2-77b261796484/volumes" Nov 03 20:22:55.056562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:55.056530 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:22:58.039651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:58.039609 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl"] Nov 03 20:22:58.045611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:58.045582 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bcc55453-fxkpl"] Nov 03 20:22:58.678798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:22:58.678764 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f96db3a-5e40-413a-8fd3-ac5eba79b2c1" path="/var/lib/kubelet/pods/9f96db3a-5e40-413a-8fd3-ac5eba79b2c1/volumes" Nov 03 20:23:06.619962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:06.619890 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7"] Nov 03 20:23:06.626053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:06.626027 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt"] Nov 03 20:23:06.626525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:06.626475 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" podUID="e2bd15bf-a127-48be-8c0d-3827037a8a06" containerName="payload-pre-processing" containerID="cri-o://427d719f7d68a02ff43e4e1661ae3cedd820aa94c9263c316ba6de9744e4ad8f" gracePeriod=30 Nov 03 20:23:06.627465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:06.627072 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" podUID="20ab9900-6823-467f-b78d-c332912d7c19" containerName="payload-processing" containerID="cri-o://e549d5734b68742f959427d3e87ffebded0473ec9063debdce8c331674590d6c" gracePeriod=30 Nov 03 20:23:07.271530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.270924 5815 generic.go:356] "Generic (PLEG): container finished" podID="e2bd15bf-a127-48be-8c0d-3827037a8a06" containerID="427d719f7d68a02ff43e4e1661ae3cedd820aa94c9263c316ba6de9744e4ad8f" exitCode=0 Nov 03 20:23:07.271530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.271011 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" event={"ID":"e2bd15bf-a127-48be-8c0d-3827037a8a06","Type":"ContainerDied","Data":"427d719f7d68a02ff43e4e1661ae3cedd820aa94c9263c316ba6de9744e4ad8f"} Nov 03 20:23:07.282597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.274710 5815 generic.go:356] "Generic (PLEG): container finished" podID="20ab9900-6823-467f-b78d-c332912d7c19" containerID="e549d5734b68742f959427d3e87ffebded0473ec9063debdce8c331674590d6c" exitCode=0 Nov 03 20:23:07.282597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.274911 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" event={"ID":"20ab9900-6823-467f-b78d-c332912d7c19","Type":"ContainerDied","Data":"e549d5734b68742f959427d3e87ffebded0473ec9063debdce8c331674590d6c"} Nov 03 20:23:07.363244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.363206 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4"] Nov 03 20:23:07.393659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.386989 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.393659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.388328 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:23:07.395277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.395252 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:23:07.400943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.400909 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-82c116-openshift-default-dockercfg-dgfpt\"" Nov 03 20:23:07.418441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.418403 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4"] Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478252 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgpr\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478312 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478345 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478389 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478429 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478539 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478568 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478592 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.482203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.478614 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.583547 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d\" (UniqueName: \"kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d\") pod \"e2bd15bf-a127-48be-8c0d-3827037a8a06\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.583662 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872\" (UniqueName: \"kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872\") pod \"20ab9900-6823-467f-b78d-c332912d7c19\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.583753 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume\") pod \"20ab9900-6823-467f-b78d-c332912d7c19\" (UID: \"20ab9900-6823-467f-b78d-c332912d7c19\") " Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.583835 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume\") pod \"e2bd15bf-a127-48be-8c0d-3827037a8a06\" (UID: \"e2bd15bf-a127-48be-8c0d-3827037a8a06\") " Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584158 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584202 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584258 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584308 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584414 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584443 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584466 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584493 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.592623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.584714 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgpr\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.616310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.616260 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d" pod "e2bd15bf-a127-48be-8c0d-3827037a8a06" (UID: "e2bd15bf-a127-48be-8c0d-3827037a8a06"). InnerVolumeSpecName "kube-api-access-b2l4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:07.626175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.626142 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.674448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.628586 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.674858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.628891 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.675005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.629139 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.675107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.635171 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgpr\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.675197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.639325 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume" pod "20ab9900-6823-467f-b78d-c332912d7c19" (UID: "20ab9900-6823-467f-b78d-c332912d7c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:07.675410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.649133 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume" pod "e2bd15bf-a127-48be-8c0d-3827037a8a06" (UID: "e2bd15bf-a127-48be-8c0d-3827037a8a06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:07.675577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.653365 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.675967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.655077 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.676088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.650918 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.680877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.680834 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872" pod "20ab9900-6823-467f-b78d-c332912d7c19" (UID: "20ab9900-6823-467f-b78d-c332912d7c19"). InnerVolumeSpecName "kube-api-access-8g872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:07.686558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.686497 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2bd15bf-a127-48be-8c0d-3827037a8a06-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.686735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.686717 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2l4d\" (UniqueName: \"kubernetes.io/projected/e2bd15bf-a127-48be-8c0d-3827037a8a06-kube-api-access-b2l4d\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.686897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.686883 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g872\" (UniqueName: \"kubernetes.io/projected/20ab9900-6823-467f-b78d-c332912d7c19-kube-api-access-8g872\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.687040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.687027 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ab9900-6823-467f-b78d-c332912d7c19-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.734613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.705104 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:07.734613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:07.727421 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:08.289381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.289294 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" Nov 03 20:23:08.289604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.289430 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7" event={"ID":"20ab9900-6823-467f-b78d-c332912d7c19","Type":"ContainerDied","Data":"4e82ee7f999d4a5c35816ae6981e05dd300e962963654100075a5cb552afbd6f"} Nov 03 20:23:08.289604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.289521 5815 scope.go:122] "RemoveContainer" containerID="e549d5734b68742f959427d3e87ffebded0473ec9063debdce8c331674590d6c" Nov 03 20:23:08.295018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.294952 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" event={"ID":"e2bd15bf-a127-48be-8c0d-3827037a8a06","Type":"ContainerDied","Data":"ce48e872681fd83e3f3425ae6bce8d25f251332af3463b7d0ccc2b9006d10f1d"} Nov 03 20:23:08.295243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.295062 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt" Nov 03 20:23:08.408571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.405799 5815 scope.go:122] "RemoveContainer" containerID="427d719f7d68a02ff43e4e1661ae3cedd820aa94c9263c316ba6de9744e4ad8f" Nov 03 20:23:08.458593 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:23:08.457031 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503031f9_bd60_4863_8b09_c43f0e56e159.slice/crio-b878ae9df420ccb2232478b09405b6ec84f750fe4410d2f19fdb71cf4d1cc068 WatchSource:0}: Error finding container b878ae9df420ccb2232478b09405b6ec84f750fe4410d2f19fdb71cf4d1cc068: Status 404 returned error can't find the container with id b878ae9df420ccb2232478b09405b6ec84f750fe4410d2f19fdb71cf4d1cc068 Nov 03 20:23:08.482607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.482574 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4"] Nov 03 20:23:08.506551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.504802 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:08.506551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.504893 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:08.506551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.504937 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:08.537769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.535440 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt"] Nov 03 20:23:08.602036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.601904 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf9b0a-8b8845fc5-khzkt"] Nov 03 20:23:08.695730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.694103 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2bd15bf-a127-48be-8c0d-3827037a8a06" path="/var/lib/kubelet/pods/e2bd15bf-a127-48be-8c0d-3827037a8a06/volumes" Nov 03 20:23:08.695730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.694604 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7"] Nov 03 20:23:08.695730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:08.694697 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf9b0a-5c6b44795d-zqrq7"] Nov 03 20:23:09.325557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.324859 5815 generic.go:356] "Generic (PLEG): container finished" podID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerID="03369573c818494f7d5d3c6e9d3094d9e1d5bcac8262f3581e5c8a6535864e7b" exitCode=0 Nov 03 20:23:09.325557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.325085 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" event={"ID":"9b80b43d-1822-4c79-8a0f-bc60123fdf41","Type":"ContainerDied","Data":"03369573c818494f7d5d3c6e9d3094d9e1d5bcac8262f3581e5c8a6535864e7b"} Nov 03 20:23:09.327854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.325944 5815 scope.go:122] "RemoveContainer" containerID="03369573c818494f7d5d3c6e9d3094d9e1d5bcac8262f3581e5c8a6535864e7b" Nov 03 20:23:09.329263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.329127 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:09.330087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.330063 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:09.330280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.330254 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:09.350219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.349240 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" event={"ID":"503031f9-bd60-4863-8b09-c43f0e56e159","Type":"ContainerStarted","Data":"180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc"} Nov 03 20:23:09.350219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.349301 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" event={"ID":"503031f9-bd60-4863-8b09-c43f0e56e159","Type":"ContainerStarted","Data":"b878ae9df420ccb2232478b09405b6ec84f750fe4410d2f19fdb71cf4d1cc068"} Nov 03 20:23:09.398548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.392463 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podStartSLOduration=2.392447774 podStartE2EDuration="2.392447774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:09.391984028 +0000 UTC m=+1298.922329702" watchObservedRunningTime="2026-11-03 20:23:09.392447774 +0000 UTC m=+1298.922793447" Nov 03 20:23:09.730787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.730756 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:09.732450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.732421 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:09.732596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:09.732489 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:10.418199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.418164 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" event={"ID":"9b80b43d-1822-4c79-8a0f-bc60123fdf41","Type":"ContainerStarted","Data":"ebff6c5a2722ad0fe8b6a6d923785cf8ccc34dba8704497f52c046ce0d8003a2"} Nov 03 20:23:10.630351 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.630304 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:10.630716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.630684 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:10.732432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.732390 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ab9900-6823-467f-b78d-c332912d7c19" path="/var/lib/kubelet/pods/20ab9900-6823-467f-b78d-c332912d7c19/volumes" Nov 03 20:23:10.733012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.732940 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:23:10.784212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.739306 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:10.784212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:10.739391 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:11.619949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:11.619913 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:11.620252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:11.620226 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:11.740075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:11.740042 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:11.740698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:11.740657 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:12.623647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:12.623614 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:12.623945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:12.623916 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:12.731532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:12.730591 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:12.731532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:12.730645 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:13.330864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.330786 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7"] Nov 03 20:23:13.331875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.331852 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e2bd15bf-a127-48be-8c0d-3827037a8a06" containerName="payload-pre-processing" Nov 03 20:23:13.332006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.331877 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bd15bf-a127-48be-8c0d-3827037a8a06" containerName="payload-pre-processing" Nov 03 20:23:13.332006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.331904 5815 cpu_manager.go:414] "containerMap: removing container" podUID="20ab9900-6823-467f-b78d-c332912d7c19" containerName="payload-processing" Nov 03 20:23:13.332006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.331912 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ab9900-6823-467f-b78d-c332912d7c19" containerName="payload-processing" Nov 03 20:23:13.332175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.332106 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bd15bf-a127-48be-8c0d-3827037a8a06" containerName="payload-pre-processing" Nov 03 20:23:13.332175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.332121 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ab9900-6823-467f-b78d-c332912d7c19" containerName="payload-processing" Nov 03 20:23:13.335964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.335944 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7"] Nov 03 20:23:13.336097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.336030 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.341705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.341684 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1b646f\"" Nov 03 20:23:13.638586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.632654 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:13.638586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.632716 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:13.759298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.757621 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.759298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.757868 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxpjl\" (UniqueName: \"kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl\") pod \"payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.797334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.770630 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:13.797334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.770703 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:13.877548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.859784 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.877548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.859978 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxpjl\" (UniqueName: \"kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl\") pod \"payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.877968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.877848 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.889194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.879993 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxpjl\" (UniqueName: \"kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl\") pod \"payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:13.953583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:13.953359 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:14.035997 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:14.035846 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:23:14.386673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.386573 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7"] Nov 03 20:23:14.426150 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:23:14.424621 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d23c89_30ec_418a_8983_9c089f4bba2d.slice/crio-4ba021878a6fa5b64b1773f7f6d513e398bbd2c60d7b2f809ae0f76dda8cb006 WatchSource:0}: Error finding container 4ba021878a6fa5b64b1773f7f6d513e398bbd2c60d7b2f809ae0f76dda8cb006: Status 404 returned error can't find the container with id 4ba021878a6fa5b64b1773f7f6d513e398bbd2c60d7b2f809ae0f76dda8cb006 Nov 03 20:23:14.640550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.637896 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:14.640550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.637965 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:14.730276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.730241 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:14.730452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.730297 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:14.844168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.844131 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" event={"ID":"80d23c89-30ec-418a-8983-9c089f4bba2d","Type":"ContainerStarted","Data":"b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b"} Nov 03 20:23:14.844342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.844198 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" event={"ID":"80d23c89-30ec-418a-8983-9c089f4bba2d","Type":"ContainerStarted","Data":"4ba021878a6fa5b64b1773f7f6d513e398bbd2c60d7b2f809ae0f76dda8cb006"} Nov 03 20:23:14.845164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.845142 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:14.889544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:14.875379 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" podStartSLOduration=1.875363318 podStartE2EDuration="1.875363318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:14.862538075 +0000 UTC m=+1304.392883746" watchObservedRunningTime="2026-11-03 20:23:14.875363318 +0000 UTC m=+1304.405708992" Nov 03 20:23:15.627537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:15.627065 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:15.627537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:15.627122 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:15.752527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:15.752301 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:23:15.752527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:15.752374 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:23:15.757496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:15.757471 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8"] Nov 03 20:23:15.758319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:15.757950 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" containerID="cri-o://7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7" gracePeriod=30 Nov 03 20:23:16.613291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:16.613243 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:23:16.613696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:16.613669 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 20:23:16.783896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:16.783859 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:16.786478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:16.786452 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:16.793779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:16.793755 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:17.617908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:17.617878 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:23:17.618434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:17.618413 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:23:17.619272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:17.619236 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:23:18.312781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.312728 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd"] Nov 03 20:23:18.432892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.432856 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.448467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.448402 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd"] Nov 03 20:23:18.453706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.453679 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9e5029-openshift-default-dockercfg-xwjkc\"" Nov 03 20:23:18.532580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.532542 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.532777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.532636 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.532777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.532746 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.532892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.532779 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.532950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.532923 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.533068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.533052 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.533137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.533085 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.533137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.533109 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhc8x\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.533302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.533283 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.634733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.634641 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.634733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.634686 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.634976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.634810 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.634976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.634851 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.634976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.634956 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635050 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635086 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635118 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhc8x\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635182 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635416 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635720 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.635864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.635725 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.636044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.636018 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.636152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.636024 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.637435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.637415 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.637731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.637713 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.650401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.650382 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.650579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.650557 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhc8x\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x\") pod \"e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:18.797045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:18.797014 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:19.172960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.172933 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd"] Nov 03 20:23:19.174340 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:23:19.174310 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922d5158_971d_45db_ad64_50fde71a4fba.slice/crio-3e7a35c78aa0caa39f82ff412308e534ca06cd2a6ffd065d4dd34d1444209e93 WatchSource:0}: Error finding container 3e7a35c78aa0caa39f82ff412308e534ca06cd2a6ffd065d4dd34d1444209e93: Status 404 returned error can't find the container with id 3e7a35c78aa0caa39f82ff412308e534ca06cd2a6ffd065d4dd34d1444209e93 Nov 03 20:23:19.178091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.178052 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:19.178181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.178140 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:19.178220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.178191 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:19.430454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.430365 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" event={"ID":"922d5158-971d-45db-ad64-50fde71a4fba","Type":"ContainerStarted","Data":"b5e4910b79ef44dd45e2fcbc4ed65e4b3a9549cee5a3684e85da5ba0848d5d1f"} Nov 03 20:23:19.430454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.430423 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" event={"ID":"922d5158-971d-45db-ad64-50fde71a4fba","Type":"ContainerStarted","Data":"3e7a35c78aa0caa39f82ff412308e534ca06cd2a6ffd065d4dd34d1444209e93"} Nov 03 20:23:19.459691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.459646 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" podStartSLOduration=2.459632428 podStartE2EDuration="2.459632428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:19.455750822 +0000 UTC m=+1308.986096519" watchObservedRunningTime="2026-11-03 20:23:19.459632428 +0000 UTC m=+1308.989978099" Nov 03 20:23:19.797377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:19.797342 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:20.805623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:20.805599 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:20.967908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:20.967873 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:23:21.083055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.082977 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083024 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083053 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083088 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083136 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083405 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:21.083500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083465 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:21.083500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083485 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:21.083658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083544 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083630 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083704 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083909 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo\") pod \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\" (UID: \"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321\") " Nov 03 20:23:21.083929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.083906 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:21.084194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.084112 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:21.084826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.084798 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.084925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.084830 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.084925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.084845 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.084925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.084859 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.084925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.084875 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.086015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.085990 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "kube-api-access-8b64w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:21.086015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.086009 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:21.086269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.086252 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:21.086598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.086579 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo" pod "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" (UID: "a5c3a2ff-efb9-48f9-ad69-2c935fbcb321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:21.186911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.186798 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.186911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.186838 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b64w\" (UniqueName: \"kubernetes.io/projected/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-kube-api-access-8b64w\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.186911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.186855 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.186911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.186868 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:21.448042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.447968 5815 generic.go:356] "Generic (PLEG): container finished" podID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerID="7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7" exitCode=0 Nov 03 20:23:21.448042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.448018 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" Nov 03 20:23:21.448218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.448054 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" event={"ID":"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321","Type":"ContainerDied","Data":"7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7"} Nov 03 20:23:21.448218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.448111 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8" event={"ID":"a5c3a2ff-efb9-48f9-ad69-2c935fbcb321","Type":"ContainerDied","Data":"2b938c4df6eb9548644a7fa512feb7d7fc6a60633b858ceaae44c104570004ae"} Nov 03 20:23:21.448218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.448141 5815 scope.go:122] "RemoveContainer" containerID="7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7" Nov 03 20:23:21.448921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.448500 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:21.449614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.449596 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:23:21.460698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.460683 5815 scope.go:122] "RemoveContainer" containerID="9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f" Nov 03 20:23:21.481823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.480692 5815 scope.go:122] "RemoveContainer" containerID="7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7" Nov 03 20:23:21.488497 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:21.487945 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7\": container with ID starting with 7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7 not found: ID does not exist" containerID="7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7" Nov 03 20:23:21.488497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.487984 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7"} err="failed to get container status \"7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7\": rpc error: code = NotFound desc = could not find container \"7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7\": container with ID starting with 7398d5f452ea953ffe7366b588b171c3ff9f0b86600231904b447149a04bb9f7 not found: ID does not exist" Nov 03 20:23:21.488497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.488009 5815 scope.go:122] "RemoveContainer" containerID="9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f" Nov 03 20:23:21.488497 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:21.488367 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f\": container with ID starting with 9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f not found: ID does not exist" containerID="9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f" Nov 03 20:23:21.488497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.488396 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f"} err="failed to get container status \"9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f\": rpc error: code = NotFound desc = could not find container \"9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f\": container with ID starting with 9137f5d19d5dd62cb189b7a73d57fd2e96630ade70f6b48623876bd1d8fd572f not found: ID does not exist" Nov 03 20:23:21.519466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.519418 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8"] Nov 03 20:23:21.521642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:21.521614 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf9b0a-openshift-default-76dc4dcdf9-7txq8"] Nov 03 20:23:22.588468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.588435 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" path="/var/lib/kubelet/pods/a5c3a2ff-efb9-48f9-ad69-2c935fbcb321/volumes" Nov 03 20:23:22.966225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.966194 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2"] Nov 03 20:23:22.966810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.966797 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" Nov 03 20:23:22.966872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.966812 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" Nov 03 20:23:22.966872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.966832 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" Nov 03 20:23:22.966872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.966838 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" Nov 03 20:23:22.966971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.966948 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" Nov 03 20:23:22.971918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.971899 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:22.984986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.984963 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9e5029\"" Nov 03 20:23:22.985597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.985545 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9e5029-dockercfg-xl4s7\"" Nov 03 20:23:22.988033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.988012 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw"] Nov 03 20:23:22.989037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.989019 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c3a2ff-efb9-48f9-ad69-2c935fbcb321" containerName="istio-proxy" Nov 03 20:23:22.998077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.993567 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:22.998077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:22.993637 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2"] Nov 03 20:23:23.004526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.001594 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw"] Nov 03 20:23:23.219593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.214886 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlc5\" (UniqueName: \"kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5\") pod \"payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.219593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.215046 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.219593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.215121 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6bkm\" (UniqueName: \"kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm\") pod \"payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.219593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.215152 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.333545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.332158 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlc5\" (UniqueName: \"kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5\") pod \"payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.333545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.332347 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.333545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.332437 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6bkm\" (UniqueName: \"kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm\") pod \"payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.333545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.332464 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.333545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.333471 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.348210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.348178 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.374400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.374356 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlc5\" (UniqueName: \"kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5\") pod \"payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.410127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.408877 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6bkm\" (UniqueName: \"kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm\") pod \"payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.593383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.593300 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:23.624846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.624823 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:23.911019 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:23:23.910988 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9425dec2_2b96_4d71_b70d_ace737399477.slice/crio-f171bdd335734923cd937c16b1bf95d22600b1479936435b8e17776e0b62c8e4 WatchSource:0}: Error finding container f171bdd335734923cd937c16b1bf95d22600b1479936435b8e17776e0b62c8e4: Status 404 returned error can't find the container with id f171bdd335734923cd937c16b1bf95d22600b1479936435b8e17776e0b62c8e4 Nov 03 20:23:23.914165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.914140 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2"] Nov 03 20:23:23.931157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:23.931132 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw"] Nov 03 20:23:24.478087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.478055 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" event={"ID":"9425dec2-2b96-4d71-b70d-ace737399477","Type":"ContainerStarted","Data":"5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6"} Nov 03 20:23:24.478306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.478122 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:24.478306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.478148 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" event={"ID":"9425dec2-2b96-4d71-b70d-ace737399477","Type":"ContainerStarted","Data":"f171bdd335734923cd937c16b1bf95d22600b1479936435b8e17776e0b62c8e4"} Nov 03 20:23:24.480682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.480651 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" event={"ID":"bbd15d14-704f-4bca-bd97-fb0cd1e8b842","Type":"ContainerStarted","Data":"af908212f9b9c284dda6e6c0c6f86eae0653d0e17cc4a5417dbc7f4041207b14"} Nov 03 20:23:24.480839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.480706 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" event={"ID":"bbd15d14-704f-4bca-bd97-fb0cd1e8b842","Type":"ContainerStarted","Data":"e8533a827ce493964c1e3bc82ed83a4d4e7ffb7c753c7230209a133bcb9ad185"} Nov 03 20:23:24.480839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.480786 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:24.505923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.505882 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" podStartSLOduration=2.505870086 podStartE2EDuration="2.505870086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:24.502280438 +0000 UTC m=+1314.032626112" watchObservedRunningTime="2026-11-03 20:23:24.505870086 +0000 UTC m=+1314.036215759" Nov 03 20:23:24.521556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:24.521488 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" podStartSLOduration=2.521474435 podStartE2EDuration="2.521474435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:24.521246567 +0000 UTC m=+1314.051592238" watchObservedRunningTime="2026-11-03 20:23:24.521474435 +0000 UTC m=+1314.051820108" Nov 03 20:23:27.086695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:27.085696 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:23:27.313533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:27.236847 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:23:28.043323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.043283 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h"] Nov 03 20:23:28.044832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.043956 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" podUID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" containerName="payload-processing" containerID="cri-o://6494d20d521b8281d0f0a90962e6aac41e3e6d3357cdf162ea42ce92343fcb0b" gracePeriod=30 Nov 03 20:23:28.052713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.052690 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj"] Nov 03 20:23:28.053130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.053084 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" podUID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" containerName="payload-pre-processing" containerID="cri-o://4e2660ff0b2d246b1249cd779ffc3743046609a8f7a436433a448fcfdf57cd8b" gracePeriod=30 Nov 03 20:23:28.351218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.350851 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" podUID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.159:9004: connect: connection refused" Nov 03 20:23:28.353373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.353300 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" podUID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.160:9004: connect: connection refused" Nov 03 20:23:28.668856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.668823 5815 generic.go:356] "Generic (PLEG): container finished" podID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" containerID="4e2660ff0b2d246b1249cd779ffc3743046609a8f7a436433a448fcfdf57cd8b" exitCode=0 Nov 03 20:23:28.668989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.668900 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" event={"ID":"1dc5ceba-3c75-4191-a691-a578b0c4efe7","Type":"ContainerDied","Data":"4e2660ff0b2d246b1249cd779ffc3743046609a8f7a436433a448fcfdf57cd8b"} Nov 03 20:23:28.668989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.668970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" event={"ID":"1dc5ceba-3c75-4191-a691-a578b0c4efe7","Type":"ContainerDied","Data":"ce0db69a352b3824f6ef31f65b835a123309010c6d0a93731fae5c539f0c1e06"} Nov 03 20:23:28.669123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.668996 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce0db69a352b3824f6ef31f65b835a123309010c6d0a93731fae5c539f0c1e06" Nov 03 20:23:28.670984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.670961 5815 generic.go:356] "Generic (PLEG): container finished" podID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" containerID="6494d20d521b8281d0f0a90962e6aac41e3e6d3357cdf162ea42ce92343fcb0b" exitCode=0 Nov 03 20:23:28.671109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.671056 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" event={"ID":"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4","Type":"ContainerDied","Data":"6494d20d521b8281d0f0a90962e6aac41e3e6d3357cdf162ea42ce92343fcb0b"} Nov 03 20:23:28.673396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.673375 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:23:28.903594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.890023 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume" pod "1dc5ceba-3c75-4191-a691-a578b0c4efe7" (UID: "1dc5ceba-3c75-4191-a691-a578b0c4efe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:28.903594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.881898 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume\") pod \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " Nov 03 20:23:28.903594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.899873 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j\" (UniqueName: \"kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j\") pod \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\" (UID: \"1dc5ceba-3c75-4191-a691-a578b0c4efe7\") " Nov 03 20:23:28.933581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.925311 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc5ceba-3c75-4191-a691-a578b0c4efe7-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:28.991538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:28.984933 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j" pod "1dc5ceba-3c75-4191-a691-a578b0c4efe7" (UID: "1dc5ceba-3c75-4191-a691-a578b0c4efe7"). InnerVolumeSpecName "kube-api-access-6ww4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:29.064559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.063275 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ww4j\" (UniqueName: \"kubernetes.io/projected/1dc5ceba-3c75-4191-a691-a578b0c4efe7-kube-api-access-6ww4j\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:29.187010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.186931 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:23:29.371930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.371899 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6\" (UniqueName: \"kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6\") pod \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " Nov 03 20:23:29.372306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.372023 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume\") pod \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\" (UID: \"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4\") " Nov 03 20:23:29.372391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.372367 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume" pod "c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" (UID: "c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:29.374263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.374235 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6" pod "c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" (UID: "c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4"). InnerVolumeSpecName "kube-api-access-k7wx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:29.473364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.473336 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:29.473364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.473359 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7wx6\" (UniqueName: \"kubernetes.io/projected/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4-kube-api-access-k7wx6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:29.679902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.679875 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" Nov 03 20:23:29.680047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.679917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h" event={"ID":"c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4","Type":"ContainerDied","Data":"8e97b79a85ba0dbd55831cd452bac1a3a637422c2f09d512ca71eb1ecfb6346f"} Nov 03 20:23:29.680047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.679980 5815 scope.go:122] "RemoveContainer" containerID="6494d20d521b8281d0f0a90962e6aac41e3e6d3357cdf162ea42ce92343fcb0b" Nov 03 20:23:29.680169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.680131 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj" Nov 03 20:23:29.814984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.814829 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h"] Nov 03 20:23:29.838179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.837396 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-5f5cd4549b-rxr5h"] Nov 03 20:23:29.844171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.844123 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj"] Nov 03 20:23:29.850219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:29.850181 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-5b9dc8d895-mpkqj"] Nov 03 20:23:30.676532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:30.670960 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" path="/var/lib/kubelet/pods/1dc5ceba-3c75-4191-a691-a578b0c4efe7/volumes" Nov 03 20:23:30.678643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:30.678613 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" path="/var/lib/kubelet/pods/c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4/volumes" Nov 03 20:23:32.548773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.548741 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k"] Nov 03 20:23:32.549815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.549707 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" containerName="payload-processing" Nov 03 20:23:32.549815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.549729 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" containerName="payload-processing" Nov 03 20:23:32.549815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.549754 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" containerName="payload-pre-processing" Nov 03 20:23:32.549815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.549762 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" containerName="payload-pre-processing" Nov 03 20:23:32.550115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.549912 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c307b8e8-e026-4aeb-837b-1e3dcb3c1cc4" containerName="payload-processing" Nov 03 20:23:32.550115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.549922 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc5ceba-3c75-4191-a691-a578b0c4efe7" containerName="payload-pre-processing" Nov 03 20:23:32.555534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.555227 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.562307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.562113 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k"] Nov 03 20:23:32.562491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.562468 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2eee8c\"/\"ipp-route-2eee8c-kserve-self-signed-certs\"" Nov 03 20:23:32.562649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.562631 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2eee8c\"/\"default-dockercfg-gb46m\"" Nov 03 20:23:32.562751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.562735 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2eee8c\"/\"kube-root-ca.crt\"" Nov 03 20:23:32.562873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.562856 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2eee8c\"/\"openshift-service-ca.crt\"" Nov 03 20:23:32.625556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.625525 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.625733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.625599 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.625733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.625689 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.625733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.625722 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxtf\" (UniqueName: \"kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.625927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.625786 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.625927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.625870 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.728421 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.728542 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.728700 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.728731 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxtf\" (UniqueName: \"kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.728832 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.728934 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.732275 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.732861 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.738549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.733095 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.773539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.770417 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.793925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.788383 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.831601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.817717 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxtf\" (UniqueName: \"kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf\") pod \"ipp-route-2eee8c-kserve-55d4959f6f-65w8k\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:32.951535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:32.935912 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:33.358092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.358057 5815 scope.go:122] "RemoveContainer" containerID="9eb03d8f25241b1298419276944ba30d2e63ddf7987f80447612e7ec43f37f2b" Nov 03 20:23:33.413829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.394963 5815 scope.go:122] "RemoveContainer" containerID="60ba2bbb1f185310a6e0dc572403092bd1b07b087df84987c719315ae47868b7" Nov 03 20:23:33.433933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.433874 5815 scope.go:122] "RemoveContainer" containerID="b39d19cb2f677a10c1eb2a8bc544f3e1fb209926d1c53ab14e78ec38f85df0c3" Nov 03 20:23:33.445325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.445303 5815 scope.go:122] "RemoveContainer" containerID="f502b0c3f9de7cc54ec3a5552fc57962235409bd8a7586027645a4dab7fb7c8e" Nov 03 20:23:33.457415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.457393 5815 scope.go:122] "RemoveContainer" containerID="ad70b2c870fc306947f8ce7757385cf95f3bc382b0a9e52354c56a8f989bf548" Nov 03 20:23:33.471189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.470792 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k"] Nov 03 20:23:33.543730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.512489 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59"] Nov 03 20:23:33.557728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.548968 5815 scope.go:122] "RemoveContainer" containerID="f319341c5d6e43455fc01146117e8a4d63828132cb3884a8670a0017806f9e46" Nov 03 20:23:33.562055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.562030 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59"] Nov 03 20:23:33.562369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.562300 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.599528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.567868 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-019ef4-openshift-default-dockercfg-rlzhd\"" Nov 03 20:23:33.640548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.625631 5815 scope.go:122] "RemoveContainer" containerID="dff89ce8d693b6a6aad22c3176133fc3b1f71a66ab8555b166ca1035392a04f3" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651320 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651411 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651611 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651647 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651675 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651714 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651775 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m86c\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651807 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.653318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.651833 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.754886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.754850 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.755218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.755192 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.755602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.755582 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.784924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.775863 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.784924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.776103 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.784924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.776320 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.786414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.786353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.786607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.786585 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.787115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.787067 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.787253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.787231 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.819371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.817931 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.819371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.818039 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m86c\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.819371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.818080 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.819371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.818107 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.855844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.855757 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.856405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.856347 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.868379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.868310 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m86c\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.924596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.923022 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token\") pod \"e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:33.952742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:33.947797 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:34.138233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:34.137830 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" event={"ID":"1b63215f-9fd3-49d3-a6c5-c1104fe44376","Type":"ContainerStarted","Data":"d6c6bc3c2200b9995c803854462bbeda663ef0a3da07b66596d732303926333d"} Nov 03 20:23:34.534926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:34.517931 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59"] Nov 03 20:23:34.534926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:34.529830 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:34.534926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:34.529999 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:34.534926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:34.530089 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:35.214051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.214011 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" event={"ID":"66e98e0c-1154-4ad3-b860-78439f986d28","Type":"ContainerStarted","Data":"3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60"} Nov 03 20:23:35.214570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.214081 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" event={"ID":"66e98e0c-1154-4ad3-b860-78439f986d28","Type":"ContainerStarted","Data":"58032a486c671358c7aad4f3a90b39e3841e9af67c63fb2c2c8c290c3cc21e93"} Nov 03 20:23:35.216769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.216739 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" event={"ID":"1b63215f-9fd3-49d3-a6c5-c1104fe44376","Type":"ContainerStarted","Data":"cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508"} Nov 03 20:23:35.236612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.236555 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podStartSLOduration=2.236496265 podStartE2EDuration="2.236496265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:35.235634904 +0000 UTC m=+1324.765980600" watchObservedRunningTime="2026-11-03 20:23:35.236496265 +0000 UTC m=+1324.766841936" Nov 03 20:23:35.258238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.258189 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podStartSLOduration=3.258172957 podStartE2EDuration="3.258172957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:35.254229551 +0000 UTC m=+1324.784575223" watchObservedRunningTime="2026-11-03 20:23:35.258172957 +0000 UTC m=+1324.788518630" Nov 03 20:23:35.549801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.549717 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:23:35.557719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.557693 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:23:35.941099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.941011 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:35.950936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.950901 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:35.956241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:35.956215 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:36.228023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:36.227987 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:36.231555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:36.231499 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:23:38.755803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.755771 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5"] Nov 03 20:23:38.760823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.760799 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:38.775726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.769075 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-019ef4-dockercfg-lw9bl\"" Nov 03 20:23:38.775726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.770565 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-019ef4\"" Nov 03 20:23:38.779025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.778954 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5"] Nov 03 20:23:38.790727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.790703 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv"] Nov 03 20:23:38.827688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.827661 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:38.835016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.834992 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv"] Nov 03 20:23:38.995537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.971999 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pxz\" (UniqueName: \"kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz\") pod \"payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:38.995975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.995945 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxc5\" (UniqueName: \"kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5\") pod \"payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:38.997000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.996976 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:38.997329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:38.997313 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:39.103580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.102737 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pxz\" (UniqueName: \"kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz\") pod \"payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:39.103580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.102842 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxc5\" (UniqueName: \"kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5\") pod \"payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:39.103580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.102897 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:39.103580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.103014 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:39.104214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.104188 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:39.104979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.104949 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:39.156535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.129817 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxc5\" (UniqueName: \"kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5\") pod \"payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:39.164681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.164649 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pxz\" (UniqueName: \"kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz\") pod \"payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:39.429678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.429600 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:39.451662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.451632 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:39.730541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.730501 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5"] Nov 03 20:23:39.732028 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:23:39.732000 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf788d31a_b296_4f14_8f1d_37c12df46426.slice/crio-ef28c229f616b96d8ed26e1c7b50cae7228284116530ca704f6cafaacc4ac474 WatchSource:0}: Error finding container ef28c229f616b96d8ed26e1c7b50cae7228284116530ca704f6cafaacc4ac474: Status 404 returned error can't find the container with id ef28c229f616b96d8ed26e1c7b50cae7228284116530ca704f6cafaacc4ac474 Nov 03 20:23:39.786931 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:23:39.786884 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ad037d_f81b_4878_b0e5_81c5db3dfbff.slice/crio-525c5c0af0f7ff9f11a52ad43601450a059cb1f9f226cdb8b25a8a17cad8f420 WatchSource:0}: Error finding container 525c5c0af0f7ff9f11a52ad43601450a059cb1f9f226cdb8b25a8a17cad8f420: Status 404 returned error can't find the container with id 525c5c0af0f7ff9f11a52ad43601450a059cb1f9f226cdb8b25a8a17cad8f420 Nov 03 20:23:39.787879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:39.787844 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv"] Nov 03 20:23:40.265652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.265614 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" event={"ID":"a7ad037d-f81b-4878-b0e5-81c5db3dfbff","Type":"ContainerStarted","Data":"45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0"} Nov 03 20:23:40.265857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.265678 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" event={"ID":"a7ad037d-f81b-4878-b0e5-81c5db3dfbff","Type":"ContainerStarted","Data":"525c5c0af0f7ff9f11a52ad43601450a059cb1f9f226cdb8b25a8a17cad8f420"} Nov 03 20:23:40.265932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.265880 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:40.267545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.267525 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" event={"ID":"f788d31a-b296-4f14-8f1d-37c12df46426","Type":"ContainerStarted","Data":"cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81"} Nov 03 20:23:40.267642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.267558 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" event={"ID":"f788d31a-b296-4f14-8f1d-37c12df46426","Type":"ContainerStarted","Data":"ef28c229f616b96d8ed26e1c7b50cae7228284116530ca704f6cafaacc4ac474"} Nov 03 20:23:40.267642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.267581 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:40.291270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.291228 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" podStartSLOduration=2.291215355 podStartE2EDuration="2.291215355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:40.288195804 +0000 UTC m=+1329.818541476" watchObservedRunningTime="2026-11-03 20:23:40.291215355 +0000 UTC m=+1329.821561028" Nov 03 20:23:40.316401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:40.316361 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" podStartSLOduration=2.316348642 podStartE2EDuration="2.316348642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:40.313987876 +0000 UTC m=+1329.844333549" watchObservedRunningTime="2026-11-03 20:23:40.316348642 +0000 UTC m=+1329.846694326" Nov 03 20:23:41.504767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:41.504720 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z"] Nov 03 20:23:41.519706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:41.519671 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9"] Nov 03 20:23:42.298126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.298092 5815 generic.go:356] "Generic (PLEG): container finished" podID="094af966-0312-4dbe-9145-5ee078f696c8" containerID="d3cb3849b62c34bc0bd23fc4755f03fa904fce2ee263a9fb15ead5b9807a9217" exitCode=0 Nov 03 20:23:42.298338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.298172 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" event={"ID":"094af966-0312-4dbe-9145-5ee078f696c8","Type":"ContainerDied","Data":"d3cb3849b62c34bc0bd23fc4755f03fa904fce2ee263a9fb15ead5b9807a9217"} Nov 03 20:23:42.298737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.298719 5815 scope.go:122] "RemoveContainer" containerID="d3cb3849b62c34bc0bd23fc4755f03fa904fce2ee263a9fb15ead5b9807a9217" Nov 03 20:23:42.300382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.300350 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:42.300478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.300421 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:42.300560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.300475 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:23:42.945533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:42.937265 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:43.262004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:43.261918 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:43.311220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:43.311184 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" event={"ID":"094af966-0312-4dbe-9145-5ee078f696c8","Type":"ContainerStarted","Data":"ed6136a6eb76bca1ccd2a4659a1d663116fd3274fe181b3d5bdb5243e48c256c"} Nov 03 20:23:43.324391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:43.324362 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:23:43.687586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:43.680846 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:23:43.687586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:43.680989 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:23:43.687586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:43.681044 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:23:43.853474 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:43.853428 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:23:44.687574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:44.675114 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:23:44.687574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:44.675172 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:23:45.605658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:45.605608 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:23:45.674946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:45.674915 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:23:45.675219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:45.675190 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:23:46.505272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:46.505231 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" podUID="f6f6baa9-816c-4229-9788-008da7b9ccd2" containerName="payload-processing" containerID="cri-o://f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1" gracePeriod=25 Nov 03 20:23:46.521045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:46.521010 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" podUID="d67aace8-4a04-47d8-a473-d99b7fa8531e" containerName="payload-pre-processing" containerID="cri-o://95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76" gracePeriod=25 Nov 03 20:23:46.689531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:46.688920 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:23:46.689531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:46.689001 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:23:47.010613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.010528 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:23:47.106742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.106714 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:23:47.131203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.131173 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume\") pod \"f6f6baa9-816c-4229-9788-008da7b9ccd2\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " Nov 03 20:23:47.131647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.131617 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume" pod "f6f6baa9-816c-4229-9788-008da7b9ccd2" (UID: "f6f6baa9-816c-4229-9788-008da7b9ccd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:47.132135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.132111 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr\" (UniqueName: \"kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr\") pod \"f6f6baa9-816c-4229-9788-008da7b9ccd2\" (UID: \"f6f6baa9-816c-4229-9788-008da7b9ccd2\") " Nov 03 20:23:47.135260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.135008 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6f6baa9-816c-4229-9788-008da7b9ccd2-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:47.135260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.135206 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr" pod "f6f6baa9-816c-4229-9788-008da7b9ccd2" (UID: "f6f6baa9-816c-4229-9788-008da7b9ccd2"). InnerVolumeSpecName "kube-api-access-bgzmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:47.236039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.236011 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume\") pod \"d67aace8-4a04-47d8-a473-d99b7fa8531e\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " Nov 03 20:23:47.236216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.236138 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d\" (UniqueName: \"kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d\") pod \"d67aace8-4a04-47d8-a473-d99b7fa8531e\" (UID: \"d67aace8-4a04-47d8-a473-d99b7fa8531e\") " Nov 03 20:23:47.236435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.236394 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume" pod "d67aace8-4a04-47d8-a473-d99b7fa8531e" (UID: "d67aace8-4a04-47d8-a473-d99b7fa8531e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:47.236604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.236588 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67aace8-4a04-47d8-a473-d99b7fa8531e-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:47.236679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.236606 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgzmr\" (UniqueName: \"kubernetes.io/projected/f6f6baa9-816c-4229-9788-008da7b9ccd2-kube-api-access-bgzmr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:47.238442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.238423 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d" pod "d67aace8-4a04-47d8-a473-d99b7fa8531e" (UID: "d67aace8-4a04-47d8-a473-d99b7fa8531e"). InnerVolumeSpecName "kube-api-access-zmn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:47.340556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.338568 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmn8d\" (UniqueName: \"kubernetes.io/projected/d67aace8-4a04-47d8-a473-d99b7fa8531e-kube-api-access-zmn8d\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:47.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.403077 5815 generic.go:356] "Generic (PLEG): container finished" podID="f6f6baa9-816c-4229-9788-008da7b9ccd2" containerID="f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1" exitCode=0 Nov 03 20:23:47.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.403284 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" event={"ID":"f6f6baa9-816c-4229-9788-008da7b9ccd2","Type":"ContainerDied","Data":"f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1"} Nov 03 20:23:47.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.403339 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" event={"ID":"f6f6baa9-816c-4229-9788-008da7b9ccd2","Type":"ContainerDied","Data":"98f488e21c892fa0943cb933453b44a18feb2942f028adad60f64f380ac16e4c"} Nov 03 20:23:47.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.403368 5815 scope.go:122] "RemoveContainer" containerID="f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1" Nov 03 20:23:47.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.403550 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z" Nov 03 20:23:47.421688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.419362 5815 generic.go:356] "Generic (PLEG): container finished" podID="d67aace8-4a04-47d8-a473-d99b7fa8531e" containerID="95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76" exitCode=0 Nov 03 20:23:47.425996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.425724 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" event={"ID":"d67aace8-4a04-47d8-a473-d99b7fa8531e","Type":"ContainerDied","Data":"95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76"} Nov 03 20:23:47.425996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.425793 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" event={"ID":"d67aace8-4a04-47d8-a473-d99b7fa8531e","Type":"ContainerDied","Data":"67d383542069e14398fcb86b3a5e40d1c86f2fca028d5df06036ac84707cdeac"} Nov 03 20:23:47.425996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.425871 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9" Nov 03 20:23:47.439962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.439784 5815 scope.go:122] "RemoveContainer" containerID="f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1" Nov 03 20:23:47.440363 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:47.440337 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1\": container with ID starting with f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1 not found: ID does not exist" containerID="f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1" Nov 03 20:23:47.440474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.440370 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1"} err="failed to get container status \"f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1\": rpc error: code = NotFound desc = could not find container \"f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1\": container with ID starting with f482d22729a0f0d9369e638c49ee7750aea1158e3dff35c8363f0eeb929afff1 not found: ID does not exist" Nov 03 20:23:47.440474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.440386 5815 scope.go:122] "RemoveContainer" containerID="95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76" Nov 03 20:23:47.455648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.455628 5815 scope.go:122] "RemoveContainer" containerID="95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76" Nov 03 20:23:47.455973 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:47.455938 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76\": container with ID starting with 95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76 not found: ID does not exist" containerID="95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76" Nov 03 20:23:47.456083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.455977 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76"} err="failed to get container status \"95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76\": rpc error: code = NotFound desc = could not find container \"95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76\": container with ID starting with 95fd06d1a9bd3ce1cb5cfe4562768d85f1a46e640613a0a33341ceb8ca80cb76 not found: ID does not exist" Nov 03 20:23:47.457798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.457778 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z"] Nov 03 20:23:47.461647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.461627 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-82c116-747858446-4c54z"] Nov 03 20:23:47.487274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.487210 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9"] Nov 03 20:23:47.504369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.504345 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-82c116-745b5997f9-v98g9"] Nov 03 20:23:47.678532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.673853 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 20:23:47.678532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:47.673916 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 20:23:48.638892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:48.638843 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67aace8-4a04-47d8-a473-d99b7fa8531e" path="/var/lib/kubelet/pods/d67aace8-4a04-47d8-a473-d99b7fa8531e/volumes" Nov 03 20:23:48.639724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:48.639684 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f6baa9-816c-4229-9788-008da7b9ccd2" path="/var/lib/kubelet/pods/f6f6baa9-816c-4229-9788-008da7b9ccd2/volumes" Nov 03 20:23:49.067536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:49.066882 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:23:49.073016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:49.072940 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:23:50.064713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:50.064683 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4"] Nov 03 20:23:50.064965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:50.064934 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" containerID="cri-o://180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc" gracePeriod=30 Nov 03 20:23:51.351575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:51.346129 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:23:51.357469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:51.356955 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:23:55.025772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.025738 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp"] Nov 03 20:23:55.029252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.029227 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6610ac02-ck5qp"] Nov 03 20:23:55.356932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.356909 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:55.456398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456367 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456420 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456467 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456490 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456567 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456657 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456786 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456800 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.456982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456929 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.456982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456947 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.456982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.456964 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr\") pod \"503031f9-bd60-4863-8b09-c43f0e56e159\" (UID: \"503031f9-bd60-4863-8b09-c43f0e56e159\") " Nov 03 20:23:55.457153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457130 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.457254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457193 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.457320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457224 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:55.457743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457712 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.457743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457728 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.457743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457738 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.457743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457747 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503031f9-bd60-4863-8b09-c43f0e56e159-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.458020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.457755 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.458979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.458953 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:55.459319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.459301 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "kube-api-access-xdgpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:55.459562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.459540 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.459562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.459549 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token" pod "503031f9-bd60-4863-8b09-c43f0e56e159" (UID: "503031f9-bd60-4863-8b09-c43f0e56e159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:55.558874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.558846 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdgpr\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-kube-api-access-xdgpr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.558874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.558869 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503031f9-bd60-4863-8b09-c43f0e56e159-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.558874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.558878 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503031f9-bd60-4863-8b09-c43f0e56e159-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.559092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.558892 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503031f9-bd60-4863-8b09-c43f0e56e159-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.599688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.599600 5815 generic.go:356] "Generic (PLEG): container finished" podID="503031f9-bd60-4863-8b09-c43f0e56e159" containerID="180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc" exitCode=0 Nov 03 20:23:55.599860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.599689 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" Nov 03 20:23:55.600018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.599980 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" event={"ID":"503031f9-bd60-4863-8b09-c43f0e56e159","Type":"ContainerDied","Data":"180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc"} Nov 03 20:23:55.600204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.600187 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4" event={"ID":"503031f9-bd60-4863-8b09-c43f0e56e159","Type":"ContainerDied","Data":"b878ae9df420ccb2232478b09405b6ec84f750fe4410d2f19fdb71cf4d1cc068"} Nov 03 20:23:55.600413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.600330 5815 scope.go:122] "RemoveContainer" containerID="180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc" Nov 03 20:23:55.631151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.631125 5815 scope.go:122] "RemoveContainer" containerID="180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc" Nov 03 20:23:55.632014 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:23:55.631944 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc\": container with ID starting with 180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc not found: ID does not exist" containerID="180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc" Nov 03 20:23:55.632014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.631988 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc"} err="failed to get container status \"180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc\": rpc error: code = NotFound desc = could not find container \"180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc\": container with ID starting with 180f043cfb654e472d5ad3c8cdc52acd10377a1359d6c75a4eac32a646b109fc not found: ID does not exist" Nov 03 20:23:55.650878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.650810 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4"] Nov 03 20:23:55.673533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:55.672730 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-88tr4"] Nov 03 20:23:56.663500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:56.663461 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" path="/var/lib/kubelet/pods/503031f9-bd60-4863-8b09-c43f0e56e159/volumes" Nov 03 20:23:56.664295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:23:56.664243 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c627dbc1-8631-42e5-8b02-9898d2f62ca4" path="/var/lib/kubelet/pods/c627dbc1-8631-42e5-8b02-9898d2f62ca4/volumes" Nov 03 20:24:02.024655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:02.024619 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8"] Nov 03 20:24:02.028317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:02.028289 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2f6661-82fh8"] Nov 03 20:24:02.597531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:02.591540 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe529a73-e2be-4675-93f6-f5aa0fa3fe64" path="/var/lib/kubelet/pods/fe529a73-e2be-4675-93f6-f5aa0fa3fe64/volumes" Nov 03 20:24:09.725199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:09.724988 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7"] Nov 03 20:24:13.972719 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:24:13.967979 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:24:14.726218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:14.726144 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" podUID="80d23c89-30ec-418a-8983-9c089f4bba2d" containerName="payload-pre-processing" containerID="cri-o://b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b" gracePeriod=25 Nov 03 20:24:14.971387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:14.971348 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67"] Nov 03 20:24:14.971685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:14.971653 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" containerID="cri-o://c7e0a81726d944564212c8ee1d5d83177f31a8d76e8eeefbd934c4c6a00bb202" gracePeriod=30 Nov 03 20:24:15.097267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.096969 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:24:15.164744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.164703 5815 generic.go:356] "Generic (PLEG): container finished" podID="80d23c89-30ec-418a-8983-9c089f4bba2d" containerID="b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b" exitCode=0 Nov 03 20:24:15.164947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.164757 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" Nov 03 20:24:15.164947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.164797 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" event={"ID":"80d23c89-30ec-418a-8983-9c089f4bba2d","Type":"ContainerDied","Data":"b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b"} Nov 03 20:24:15.164947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.164892 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7" event={"ID":"80d23c89-30ec-418a-8983-9c089f4bba2d","Type":"ContainerDied","Data":"4ba021878a6fa5b64b1773f7f6d513e398bbd2c60d7b2f809ae0f76dda8cb006"} Nov 03 20:24:15.165128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.164962 5815 scope.go:122] "RemoveContainer" containerID="b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b" Nov 03 20:24:15.179076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.179055 5815 scope.go:122] "RemoveContainer" containerID="b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b" Nov 03 20:24:15.179412 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:24:15.179381 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b\": container with ID starting with b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b not found: ID does not exist" containerID="b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b" Nov 03 20:24:15.179588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.179423 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b"} err="failed to get container status \"b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b\": rpc error: code = NotFound desc = could not find container \"b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b\": container with ID starting with b4d813be4dd6eb6bea9ccc180ea91e541fd8478b66177f23bac4f1582d1ca84b not found: ID does not exist" Nov 03 20:24:15.204337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.204302 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume\") pod \"80d23c89-30ec-418a-8983-9c089f4bba2d\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " Nov 03 20:24:15.204535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.204492 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl\" (UniqueName: \"kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl\") pod \"80d23c89-30ec-418a-8983-9c089f4bba2d\" (UID: \"80d23c89-30ec-418a-8983-9c089f4bba2d\") " Nov 03 20:24:15.204724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.204651 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume" pod "80d23c89-30ec-418a-8983-9c089f4bba2d" (UID: "80d23c89-30ec-418a-8983-9c089f4bba2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:15.205053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.205033 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80d23c89-30ec-418a-8983-9c089f4bba2d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:15.206896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.206869 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl" pod "80d23c89-30ec-418a-8983-9c089f4bba2d" (UID: "80d23c89-30ec-418a-8983-9c089f4bba2d"). InnerVolumeSpecName "kube-api-access-rxpjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:15.253496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.253471 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:24:15.306608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.306544 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxpjl\" (UniqueName: \"kubernetes.io/projected/80d23c89-30ec-418a-8983-9c089f4bba2d-kube-api-access-rxpjl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:15.566585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.562562 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7"] Nov 03 20:24:15.589581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:15.587396 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1b646f-6d7d8c7847-lgfn7"] Nov 03 20:24:16.360982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:16.360950 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg"] Nov 03 20:24:16.361488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:16.361203 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" containerID="cri-o://ed6136a6eb76bca1ccd2a4659a1d663116fd3274fe181b3d5bdb5243e48c256c" gracePeriod=30 Nov 03 20:24:16.638239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:16.638157 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d23c89-30ec-418a-8983-9c089f4bba2d" path="/var/lib/kubelet/pods/80d23c89-30ec-418a-8983-9c089f4bba2d/volumes" Nov 03 20:24:18.099758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.099728 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp"] Nov 03 20:24:18.100918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100892 5815 cpu_manager.go:414] "containerMap: removing container" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" Nov 03 20:24:18.100918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100913 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" Nov 03 20:24:18.100918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100926 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f6f6baa9-816c-4229-9788-008da7b9ccd2" containerName="payload-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100931 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f6baa9-816c-4229-9788-008da7b9ccd2" containerName="payload-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100957 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d67aace8-4a04-47d8-a473-d99b7fa8531e" containerName="payload-pre-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100962 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67aace8-4a04-47d8-a473-d99b7fa8531e" containerName="payload-pre-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100976 5815 cpu_manager.go:414] "containerMap: removing container" podUID="80d23c89-30ec-418a-8983-9c089f4bba2d" containerName="payload-pre-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.100981 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d23c89-30ec-418a-8983-9c089f4bba2d" containerName="payload-pre-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.101058 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="503031f9-bd60-4863-8b09-c43f0e56e159" containerName="istio-proxy" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.101064 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f6baa9-816c-4229-9788-008da7b9ccd2" containerName="payload-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.101076 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67aace8-4a04-47d8-a473-d99b7fa8531e" containerName="payload-pre-processing" Nov 03 20:24:18.101109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.101083 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d23c89-30ec-418a-8983-9c089f4bba2d" containerName="payload-pre-processing" Nov 03 20:24:18.104693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.104675 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.110967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.110943 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp"] Nov 03 20:24:18.139403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.138712 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:18.139403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.138795 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:24:18.245414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.245382 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsw7\" (UniqueName: \"kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.245691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.245656 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.245750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.245719 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.353649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.353560 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsw7\" (UniqueName: \"kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.358793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.358757 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.369721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.359887 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.369955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.369689 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.441724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.441102 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.441724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.441349 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsw7\" (UniqueName: \"kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7\") pod \"maas-api-revoke-sub-837a9a5a-tvbnp\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:18.726043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:18.726014 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:19.050152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:19.050126 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp"] Nov 03 20:24:19.050935 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:24:19.050913 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6f54ca_2615_4dad_b55c_bb03e957ff47.slice/crio-52f2ed257a1541e99f63d0358211688c92aa66e4b96a042151c1165f3ef63623 WatchSource:0}: Error finding container 52f2ed257a1541e99f63d0358211688c92aa66e4b96a042151c1165f3ef63623: Status 404 returned error can't find the container with id 52f2ed257a1541e99f63d0358211688c92aa66e4b96a042151c1165f3ef63623 Nov 03 20:24:19.096535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:19.088785 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:24:19.096535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:19.088841 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:24:19.445125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:19.445089 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" event={"ID":"9f6f54ca-2615-4dad-b55c-bb03e957ff47","Type":"ContainerStarted","Data":"52f2ed257a1541e99f63d0358211688c92aa66e4b96a042151c1165f3ef63623"} Nov 03 20:24:20.715594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.712442 5815 generic.go:356] "Generic (PLEG): container finished" podID="9f6f54ca-2615-4dad-b55c-bb03e957ff47" containerID="abba51966699ec974c5afc477c396e8052d2b0be97a2d2e2ea7a9b44fbde94d2" exitCode=0 Nov 03 20:24:20.715594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.712647 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" event={"ID":"9f6f54ca-2615-4dad-b55c-bb03e957ff47","Type":"ContainerDied","Data":"abba51966699ec974c5afc477c396e8052d2b0be97a2d2e2ea7a9b44fbde94d2"} Nov 03 20:24:20.717480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.717446 5815 generic.go:356] "Generic (PLEG): container finished" podID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerID="c7e0a81726d944564212c8ee1d5d83177f31a8d76e8eeefbd934c4c6a00bb202" exitCode=0 Nov 03 20:24:20.718023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.717740 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" event={"ID":"0ac25f1d-3097-426f-a9be-c5c1baaacb3e","Type":"ContainerDied","Data":"c7e0a81726d944564212c8ee1d5d83177f31a8d76e8eeefbd934c4c6a00bb202"} Nov 03 20:24:20.719552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.719535 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:24:20.806065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.806024 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.806327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.806307 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.806472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.806459 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.806641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.806627 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.809700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.809670 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.810039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.810021 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.810700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.810680 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.810886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.810851 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.811053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.811037 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token\") pod \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\" (UID: \"0ac25f1d-3097-426f-a9be-c5c1baaacb3e\") " Nov 03 20:24:20.815177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.815153 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:20.817995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.816280 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:20.817995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.816332 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:20.817995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.816765 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:20.818322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.818305 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:20.818461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.818420 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:20.819968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.819935 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "kube-api-access-fthmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:20.822597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.821139 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:20.822906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.822882 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy" pod "0ac25f1d-3097-426f-a9be-c5c1baaacb3e" (UID: "0ac25f1d-3097-426f-a9be-c5c1baaacb3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.912924 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.912961 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.912976 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.912989 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.913003 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fthmx\" (UniqueName: \"kubernetes.io/projected/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-kube-api-access-fthmx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.913022 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.913037 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.913050 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:20.913093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:20.913064 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ac25f1d-3097-426f-a9be-c5c1baaacb3e-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:21.833584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.832247 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" event={"ID":"0ac25f1d-3097-426f-a9be-c5c1baaacb3e","Type":"ContainerDied","Data":"f87450aecaf717c2940bb78b1270292e8f808a1f067b1ac03579a187d872e736"} Nov 03 20:24:21.833584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.832317 5815 scope.go:122] "RemoveContainer" containerID="c7e0a81726d944564212c8ee1d5d83177f31a8d76e8eeefbd934c4c6a00bb202" Nov 03 20:24:21.833584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.832441 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67" Nov 03 20:24:21.840111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.838474 5815 generic.go:356] "Generic (PLEG): container finished" podID="094af966-0312-4dbe-9145-5ee078f696c8" containerID="ed6136a6eb76bca1ccd2a4659a1d663116fd3274fe181b3d5bdb5243e48c256c" exitCode=0 Nov 03 20:24:21.840111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.838891 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" event={"ID":"094af966-0312-4dbe-9145-5ee078f696c8","Type":"ContainerDied","Data":"ed6136a6eb76bca1ccd2a4659a1d663116fd3274fe181b3d5bdb5243e48c256c"} Nov 03 20:24:21.864726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.864278 5815 scope.go:122] "RemoveContainer" containerID="d3cb3849b62c34bc0bd23fc4755f03fa904fce2ee263a9fb15ead5b9807a9217" Nov 03 20:24:21.916160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.916131 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67"] Nov 03 20:24:21.942301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:21.942236 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-9kx67"] Nov 03 20:24:22.028143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.027365 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:24:22.134247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.134204 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135002 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135083 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135120 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135227 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135311 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135490 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.135676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135652 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:22.135946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135735 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.136022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135893 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:22.136022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135946 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:22.136022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.135972 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data\") pod \"094af966-0312-4dbe-9145-5ee078f696c8\" (UID: \"094af966-0312-4dbe-9145-5ee078f696c8\") " Nov 03 20:24:22.136251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.136231 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:22.136665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.136631 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:22.136752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.136665 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.136752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.136686 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.136752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.136699 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.136752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.136711 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.137158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.137131 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:22.138734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.138711 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:22.139194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.139170 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:22.139761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.139742 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2" pod "094af966-0312-4dbe-9145-5ee078f696c8" (UID: "094af966-0312-4dbe-9145-5ee078f696c8"). InnerVolumeSpecName "kube-api-access-rddg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:22.160497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.160480 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:22.238229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.238196 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094af966-0312-4dbe-9145-5ee078f696c8-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.238433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.238420 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094af966-0312-4dbe-9145-5ee078f696c8-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.238572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.238549 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rddg2\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-kube-api-access-rddg2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.238733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.238719 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094af966-0312-4dbe-9145-5ee078f696c8-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.238874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.238859 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094af966-0312-4dbe-9145-5ee078f696c8-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.340588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.339719 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7\" (UniqueName: \"kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7\") pod \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " Nov 03 20:24:22.340588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.340259 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca\") pod \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " Nov 03 20:24:22.340588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.340393 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca\") pod \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\" (UID: \"9f6f54ca-2615-4dad-b55c-bb03e957ff47\") " Nov 03 20:24:22.354564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.352244 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7" pod "9f6f54ca-2615-4dad-b55c-bb03e957ff47" (UID: "9f6f54ca-2615-4dad-b55c-bb03e957ff47"). InnerVolumeSpecName "kube-api-access-fnsw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:22.363409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.363107 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca" pod "9f6f54ca-2615-4dad-b55c-bb03e957ff47" (UID: "9f6f54ca-2615-4dad-b55c-bb03e957ff47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:22.363751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.363670 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca" pod "9f6f54ca-2615-4dad-b55c-bb03e957ff47" (UID: "9f6f54ca-2615-4dad-b55c-bb03e957ff47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:22.442325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.442294 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.442325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.442324 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f6f54ca-2615-4dad-b55c-bb03e957ff47-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.442563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.442335 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsw7\" (UniqueName: \"kubernetes.io/projected/9f6f54ca-2615-4dad-b55c-bb03e957ff47-kube-api-access-fnsw7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:22.593564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.593456 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" path="/var/lib/kubelet/pods/0ac25f1d-3097-426f-a9be-c5c1baaacb3e/volumes" Nov 03 20:24:22.849547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.849434 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" Nov 03 20:24:22.849547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.849445 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp" event={"ID":"9f6f54ca-2615-4dad-b55c-bb03e957ff47","Type":"ContainerDied","Data":"52f2ed257a1541e99f63d0358211688c92aa66e4b96a042151c1165f3ef63623"} Nov 03 20:24:22.849547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.849500 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f2ed257a1541e99f63d0358211688c92aa66e4b96a042151c1165f3ef63623" Nov 03 20:24:22.852361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.852328 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" Nov 03 20:24:22.852491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.852391 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg" event={"ID":"094af966-0312-4dbe-9145-5ee078f696c8","Type":"ContainerDied","Data":"e1d527e04983e25d05142f2b832b1071d675889b4a7b413d8a2b320b461247cb"} Nov 03 20:24:22.852491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.852445 5815 scope.go:122] "RemoveContainer" containerID="ed6136a6eb76bca1ccd2a4659a1d663116fd3274fe181b3d5bdb5243e48c256c" Nov 03 20:24:22.874941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.874915 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg"] Nov 03 20:24:22.880759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:22.880738 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a596f7-openshift-default-d48f46df-qxkpg"] Nov 03 20:24:23.097066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:23.097031 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw"] Nov 03 20:24:23.112239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:23.112157 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2"] Nov 03 20:24:24.612278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:24.612245 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094af966-0312-4dbe-9145-5ee078f696c8" path="/var/lib/kubelet/pods/094af966-0312-4dbe-9145-5ee078f696c8/volumes" Nov 03 20:24:28.097844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.097775 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" podUID="bbd15d14-704f-4bca-bd97-fb0cd1e8b842" containerName="payload-processing" containerID="cri-o://af908212f9b9c284dda6e6c0c6f86eae0653d0e17cc4a5417dbc7f4041207b14" gracePeriod=25 Nov 03 20:24:28.113655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.113455 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" podUID="9425dec2-2b96-4d71-b70d-ace737399477" containerName="payload-pre-processing" containerID="cri-o://5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6" gracePeriod=25 Nov 03 20:24:28.267785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.267747 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck"] Nov 03 20:24:28.268982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.268955 5815 cpu_manager.go:414] "containerMap: removing container" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" Nov 03 20:24:28.269155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269140 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" Nov 03 20:24:28.269318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269305 5815 cpu_manager.go:414] "containerMap: removing container" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" Nov 03 20:24:28.269416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269402 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" Nov 03 20:24:28.269539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269525 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" Nov 03 20:24:28.269651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269639 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" Nov 03 20:24:28.269761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269751 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9f6f54ca-2615-4dad-b55c-bb03e957ff47" containerName="revoke-keys" Nov 03 20:24:28.269841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.269831 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f6f54ca-2615-4dad-b55c-bb03e957ff47" containerName="revoke-keys" Nov 03 20:24:28.270090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.270076 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" Nov 03 20:24:28.270188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.270178 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac25f1d-3097-426f-a9be-c5c1baaacb3e" containerName="istio-proxy" Nov 03 20:24:28.270276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.270266 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f6f54ca-2615-4dad-b55c-bb03e957ff47" containerName="revoke-keys" Nov 03 20:24:28.270362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.270353 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="094af966-0312-4dbe-9145-5ee078f696c8" containerName="istio-proxy" Nov 03 20:24:28.279602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.274791 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.304430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.304363 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2eee8c-openshift-default-dockercfg-q5wm8\"" Nov 03 20:24:28.308450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.308239 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck"] Nov 03 20:24:28.355202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355156 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355343 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355446 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355498 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355589 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355624 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355712 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355799 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.355875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.355822 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76j7t\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.457769 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.457828 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.457862 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.457923 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.457948 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.458005 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.458067 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.458091 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76j7t\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.458548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.458139 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.460074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.459812 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.461300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.460974 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.461300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.461110 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.461300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.461255 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.462080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.462008 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.497544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.486439 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.497544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.489606 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76j7t\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.497544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.490084 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.529769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.522762 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.694431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.694095 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:28.929191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:28.928916 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:24:29.075456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.074107 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume\") pod \"9425dec2-2b96-4d71-b70d-ace737399477\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " Nov 03 20:24:29.075456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.074283 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm\" (UniqueName: \"kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm\") pod \"9425dec2-2b96-4d71-b70d-ace737399477\" (UID: \"9425dec2-2b96-4d71-b70d-ace737399477\") " Nov 03 20:24:29.076254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.076177 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume" pod "9425dec2-2b96-4d71-b70d-ace737399477" (UID: "9425dec2-2b96-4d71-b70d-ace737399477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:29.189544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.142265 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm" pod "9425dec2-2b96-4d71-b70d-ace737399477" (UID: "9425dec2-2b96-4d71-b70d-ace737399477"). InnerVolumeSpecName "kube-api-access-k6bkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:29.207721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.201012 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9425dec2-2b96-4d71-b70d-ace737399477-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:29.207721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.201045 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6bkm\" (UniqueName: \"kubernetes.io/projected/9425dec2-2b96-4d71-b70d-ace737399477-kube-api-access-k6bkm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:29.273592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.263445 5815 generic.go:356] "Generic (PLEG): container finished" podID="9425dec2-2b96-4d71-b70d-ace737399477" containerID="5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6" exitCode=0 Nov 03 20:24:29.273592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.265735 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" event={"ID":"9425dec2-2b96-4d71-b70d-ace737399477","Type":"ContainerDied","Data":"5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6"} Nov 03 20:24:29.273592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.265802 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" event={"ID":"9425dec2-2b96-4d71-b70d-ace737399477","Type":"ContainerDied","Data":"f171bdd335734923cd937c16b1bf95d22600b1479936435b8e17776e0b62c8e4"} Nov 03 20:24:29.273592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.265840 5815 scope.go:122] "RemoveContainer" containerID="5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6" Nov 03 20:24:29.273592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.266025 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2" Nov 03 20:24:29.302331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.302094 5815 generic.go:356] "Generic (PLEG): container finished" podID="bbd15d14-704f-4bca-bd97-fb0cd1e8b842" containerID="af908212f9b9c284dda6e6c0c6f86eae0653d0e17cc4a5417dbc7f4041207b14" exitCode=0 Nov 03 20:24:29.302331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.302238 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" event={"ID":"bbd15d14-704f-4bca-bd97-fb0cd1e8b842","Type":"ContainerDied","Data":"af908212f9b9c284dda6e6c0c6f86eae0653d0e17cc4a5417dbc7f4041207b14"} Nov 03 20:24:29.421454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.421357 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:24:29.447763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.434872 5815 scope.go:122] "RemoveContainer" containerID="5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6" Nov 03 20:24:29.447763 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:24:29.435270 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6\": container with ID starting with 5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6 not found: ID does not exist" containerID="5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6" Nov 03 20:24:29.447763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.435302 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6"} err="failed to get container status \"5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6\": rpc error: code = NotFound desc = could not find container \"5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6\": container with ID starting with 5638f9ced6980891c8672b858dc68eee9dd8ba600c5f4c4774767d9ff21b31c6 not found: ID does not exist" Nov 03 20:24:29.447763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.438568 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2"] Nov 03 20:24:29.447763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.441127 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9e5029-c4df44dff-9q8z2"] Nov 03 20:24:29.449734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.449704 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck"] Nov 03 20:24:29.452443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.452405 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:24:29.452532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.452487 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:24:29.452579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.452558 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:24:29.519449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.519408 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume\") pod \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " Nov 03 20:24:29.519662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.519478 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5\" (UniqueName: \"kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5\") pod \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\" (UID: \"bbd15d14-704f-4bca-bd97-fb0cd1e8b842\") " Nov 03 20:24:29.521834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.520125 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume" pod "bbd15d14-704f-4bca-bd97-fb0cd1e8b842" (UID: "bbd15d14-704f-4bca-bd97-fb0cd1e8b842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:29.521834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.520840 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:29.535550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.535485 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5" pod "bbd15d14-704f-4bca-bd97-fb0cd1e8b842" (UID: "bbd15d14-704f-4bca-bd97-fb0cd1e8b842"). InnerVolumeSpecName "kube-api-access-9dlc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:29.622861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:29.622824 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dlc5\" (UniqueName: \"kubernetes.io/projected/bbd15d14-704f-4bca-bd97-fb0cd1e8b842-kube-api-access-9dlc5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:30.365623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.363611 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" event={"ID":"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc","Type":"ContainerStarted","Data":"df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc"} Nov 03 20:24:30.365623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.363683 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" event={"ID":"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc","Type":"ContainerStarted","Data":"e1d7e79086e72d0c28f0ae026278d66d6de91d7dd4aaf4e57b3c63fbe6e6f6d5"} Nov 03 20:24:30.385487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.383845 5815 generic.go:356] "Generic (PLEG): container finished" podID="66e98e0c-1154-4ad3-b860-78439f986d28" containerID="3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60" exitCode=0 Nov 03 20:24:30.385487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.383956 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" event={"ID":"66e98e0c-1154-4ad3-b860-78439f986d28","Type":"ContainerDied","Data":"3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60"} Nov 03 20:24:30.385487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.384401 5815 scope.go:122] "RemoveContainer" containerID="3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60" Nov 03 20:24:30.390531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.387317 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:24:30.390818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.390788 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:24:30.390969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.390951 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:24:30.439077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.439042 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" event={"ID":"bbd15d14-704f-4bca-bd97-fb0cd1e8b842","Type":"ContainerDied","Data":"e8533a827ce493964c1e3bc82ed83a4d4e7ffb7c753c7230209a133bcb9ad185"} Nov 03 20:24:30.439255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.439114 5815 scope.go:122] "RemoveContainer" containerID="af908212f9b9c284dda6e6c0c6f86eae0653d0e17cc4a5417dbc7f4041207b14" Nov 03 20:24:30.439327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.439303 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw" Nov 03 20:24:30.500303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.500243 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podStartSLOduration=2.5002284489999997 podStartE2EDuration="2.500228449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:30.413935209 +0000 UTC m=+1379.944280888" watchObservedRunningTime="2026-11-03 20:24:30.500228449 +0000 UTC m=+1380.030574120" Nov 03 20:24:30.523095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.523063 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw"] Nov 03 20:24:30.531623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.531598 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e5029-7df69b99fd-4qsmw"] Nov 03 20:24:30.630663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.630582 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9425dec2-2b96-4d71-b70d-ace737399477" path="/var/lib/kubelet/pods/9425dec2-2b96-4d71-b70d-ace737399477/volumes" Nov 03 20:24:30.631053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.631032 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd15d14-704f-4bca-bd97-fb0cd1e8b842" path="/var/lib/kubelet/pods/bbd15d14-704f-4bca-bd97-fb0cd1e8b842/volumes" Nov 03 20:24:30.701462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.699649 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:30.701462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.700951 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:24:30.701462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:30.701017 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:24:31.050633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.050605 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn"] Nov 03 20:24:31.051245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.051227 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bbd15d14-704f-4bca-bd97-fb0cd1e8b842" containerName="payload-processing" Nov 03 20:24:31.051348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.051248 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd15d14-704f-4bca-bd97-fb0cd1e8b842" containerName="payload-processing" Nov 03 20:24:31.051348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.051275 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9425dec2-2b96-4d71-b70d-ace737399477" containerName="payload-pre-processing" Nov 03 20:24:31.051348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.051282 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9425dec2-2b96-4d71-b70d-ace737399477" containerName="payload-pre-processing" Nov 03 20:24:31.051547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.051455 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9425dec2-2b96-4d71-b70d-ace737399477" containerName="payload-pre-processing" Nov 03 20:24:31.051547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.051482 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd15d14-704f-4bca-bd97-fb0cd1e8b842" containerName="payload-processing" Nov 03 20:24:31.056150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.056130 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.105700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.105668 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn"] Nov 03 20:24:31.112407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.112189 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:31.112407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.112364 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:24:31.158184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.158143 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6bt\" (UniqueName: \"kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.158355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.158217 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.158355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.158251 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.260404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.260367 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6bt\" (UniqueName: \"kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.260621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.260436 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.260621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.260469 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.261253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.261225 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.263093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.263070 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.326577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.304243 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6bt\" (UniqueName: \"kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt\") pod \"maas-api-revoke-sub-2409989a-t62fn\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.404538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.401586 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:31.485335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.483455 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" event={"ID":"66e98e0c-1154-4ad3-b860-78439f986d28","Type":"ContainerStarted","Data":"575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed"} Nov 03 20:24:31.740971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.740648 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:24:31.740971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.740705 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:24:31.852302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.852272 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn"] Nov 03 20:24:31.902686 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:24:31.902642 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27accbd_71f6_441a_bc62_9f4e1aa53f4b.slice/crio-e3845634bdd20d897baf17ccf3720903be2d157f05898d18068a10b26e2081ad WatchSource:0}: Error finding container e3845634bdd20d897baf17ccf3720903be2d157f05898d18068a10b26e2081ad: Status 404 returned error can't find the container with id e3845634bdd20d897baf17ccf3720903be2d157f05898d18068a10b26e2081ad Nov 03 20:24:31.951309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.951260 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:24:31.952080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.952030 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:24:31.952157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:31.952077 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:24:32.140588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.140227 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd"] Nov 03 20:24:32.140588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.140469 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" podUID="922d5158-971d-45db-ad64-50fde71a4fba" containerName="istio-proxy" containerID="cri-o://b5e4910b79ef44dd45e2fcbc4ed65e4b3a9549cee5a3684e85da5ba0848d5d1f" gracePeriod=30 Nov 03 20:24:32.650147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.650098 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" event={"ID":"b27accbd-71f6-441a-bc62-9f4e1aa53f4b","Type":"ContainerStarted","Data":"e3845634bdd20d897baf17ccf3720903be2d157f05898d18068a10b26e2081ad"} Nov 03 20:24:32.700972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.700934 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:24:32.701678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.701642 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:24:32.951472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.951393 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:24:32.951472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:32.951442 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:24:33.664757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:33.664114 5815 generic.go:356] "Generic (PLEG): container finished" podID="b27accbd-71f6-441a-bc62-9f4e1aa53f4b" containerID="216d983378d96dac70b963ac4d103d740fbbc0cccbe101836a25cd4e68510130" exitCode=0 Nov 03 20:24:33.664757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:33.664325 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" event={"ID":"b27accbd-71f6-441a-bc62-9f4e1aa53f4b","Type":"ContainerDied","Data":"216d983378d96dac70b963ac4d103d740fbbc0cccbe101836a25cd4e68510130"} Nov 03 20:24:33.719523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:33.719461 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:24:33.719725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:33.719555 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:24:33.951702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:33.951615 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:24:33.951702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:33.951669 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:24:34.131535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.130860 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7"] Nov 03 20:24:34.176532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.161205 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.259751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.259580 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7"] Nov 03 20:24:34.348929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.348620 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzm8\" (UniqueName: \"kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.348929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.348843 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.348929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.348903 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.349964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.349943 5815 scope.go:122] "RemoveContainer" containerID="67a8a799efe3b54c30281f590c19663639f6539ce6f60036c6df57a5753c78ab" Nov 03 20:24:34.362389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.362364 5815 scope.go:122] "RemoveContainer" containerID="f456d1cbd6b6ef4864c83f02148325ef19023fbe1533e886379cc1aa21722b60" Nov 03 20:24:34.434002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.433979 5815 scope.go:122] "RemoveContainer" containerID="ae9357d3e2c222b26e5236d31310a47eb105fd79f598194374e26ddc9f1d3a7b" Nov 03 20:24:34.450064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.450036 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.450200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.450160 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzm8\" (UniqueName: \"kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.450306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.450289 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.451199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.451174 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.453031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.453003 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.462956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.462935 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzm8\" (UniqueName: \"kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8\") pod \"maas-api-revoke-sub-15094ed4-qqcx7\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.516607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.514949 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:34.701999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.699932 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:24:34.701999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.699990 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:24:34.951473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.951066 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:24:34.951473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:34.951118 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:24:35.156205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.152859 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7"] Nov 03 20:24:35.524532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.524396 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:35.743622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.689026 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca\") pod \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " Nov 03 20:24:35.743622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.689100 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca\") pod \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " Nov 03 20:24:35.743622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.689267 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt\" (UniqueName: \"kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt\") pod \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\" (UID: \"b27accbd-71f6-441a-bc62-9f4e1aa53f4b\") " Nov 03 20:24:35.743622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.690107 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca" pod "b27accbd-71f6-441a-bc62-9f4e1aa53f4b" (UID: "b27accbd-71f6-441a-bc62-9f4e1aa53f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:35.743622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.695022 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca" pod "b27accbd-71f6-441a-bc62-9f4e1aa53f4b" (UID: "b27accbd-71f6-441a-bc62-9f4e1aa53f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:35.763031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.746829 5815 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Nov 03 20:24:35.763031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.746898 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Nov 03 20:24:35.763384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.763325 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt" pod "b27accbd-71f6-441a-bc62-9f4e1aa53f4b" (UID: "b27accbd-71f6-441a-bc62-9f4e1aa53f4b"). InnerVolumeSpecName "kube-api-access-mc6bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:35.822531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.798348 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:35.822837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.822811 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:35.822955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.822942 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc6bt\" (UniqueName: \"kubernetes.io/projected/b27accbd-71f6-441a-bc62-9f4e1aa53f4b-kube-api-access-mc6bt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:35.953818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.953295 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:24:35.953818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.953362 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:24:35.977537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.976138 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" event={"ID":"b27accbd-71f6-441a-bc62-9f4e1aa53f4b","Type":"ContainerDied","Data":"e3845634bdd20d897baf17ccf3720903be2d157f05898d18068a10b26e2081ad"} Nov 03 20:24:35.977537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.976231 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3845634bdd20d897baf17ccf3720903be2d157f05898d18068a10b26e2081ad" Nov 03 20:24:35.977537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.976301 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn" Nov 03 20:24:35.990531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.988797 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" event={"ID":"1b4351ba-b1cb-467d-ab0b-7158f7164dcc","Type":"ContainerStarted","Data":"13d64391b914d6e752eecb6240bc37659139ca59be9f020543dce79ff529f36b"} Nov 03 20:24:35.990531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:35.988862 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" event={"ID":"1b4351ba-b1cb-467d-ab0b-7158f7164dcc","Type":"ContainerStarted","Data":"4f14bab9499cbdb222ce6857df3f0207d24a1df029d8964954b7c36e98e2d612"} Nov 03 20:24:36.041803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:36.041706 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" podStartSLOduration=2.041691567 podStartE2EDuration="2.041691567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:36.039937679 +0000 UTC m=+1385.570283351" watchObservedRunningTime="2026-11-03 20:24:36.041691567 +0000 UTC m=+1385.572037249" Nov 03 20:24:36.235533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:36.234656 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:24:36.951124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:36.951084 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:24:36.951543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:36.951141 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:24:37.011527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.011475 5815 generic.go:356] "Generic (PLEG): container finished" podID="1b4351ba-b1cb-467d-ab0b-7158f7164dcc" containerID="13d64391b914d6e752eecb6240bc37659139ca59be9f020543dce79ff529f36b" exitCode=0 Nov 03 20:24:37.011832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.011813 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" event={"ID":"1b4351ba-b1cb-467d-ab0b-7158f7164dcc","Type":"ContainerDied","Data":"13d64391b914d6e752eecb6240bc37659139ca59be9f020543dce79ff529f36b"} Nov 03 20:24:37.025027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.024994 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:37.025741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.025369 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:37.027432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.027388 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:24:37.490685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.490646 5815 patch_prober.go:29] interesting pod/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:24:37.490843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.490717 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" podUID="922d5158-971d-45db-ad64-50fde71a4fba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:37.956896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.956811 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:24:37.958024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:37.958002 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:24:38.233482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.233443 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:38.264835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.264805 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca\") pod \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " Nov 03 20:24:38.264996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.264848 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca\") pod \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " Nov 03 20:24:38.265125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.265106 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8\" (UniqueName: \"kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8\") pod \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\" (UID: \"1b4351ba-b1cb-467d-ab0b-7158f7164dcc\") " Nov 03 20:24:38.265290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.265270 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca" pod "1b4351ba-b1cb-467d-ab0b-7158f7164dcc" (UID: "1b4351ba-b1cb-467d-ab0b-7158f7164dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:38.265635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.265616 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:38.267924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.267895 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca" pod "1b4351ba-b1cb-467d-ab0b-7158f7164dcc" (UID: "1b4351ba-b1cb-467d-ab0b-7158f7164dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:38.268047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.267998 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8" pod "1b4351ba-b1cb-467d-ab0b-7158f7164dcc" (UID: "1b4351ba-b1cb-467d-ab0b-7158f7164dcc"). InnerVolumeSpecName "kube-api-access-8gzm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:38.367532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.366933 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzm8\" (UniqueName: \"kubernetes.io/projected/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-kube-api-access-8gzm8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:38.367532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:38.366963 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b4351ba-b1cb-467d-ab0b-7158f7164dcc-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.036126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.036091 5815 generic.go:356] "Generic (PLEG): container finished" podID="922d5158-971d-45db-ad64-50fde71a4fba" containerID="b5e4910b79ef44dd45e2fcbc4ed65e4b3a9549cee5a3684e85da5ba0848d5d1f" exitCode=0 Nov 03 20:24:39.036661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.036157 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" event={"ID":"922d5158-971d-45db-ad64-50fde71a4fba","Type":"ContainerDied","Data":"b5e4910b79ef44dd45e2fcbc4ed65e4b3a9549cee5a3684e85da5ba0848d5d1f"} Nov 03 20:24:39.038244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.038222 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" Nov 03 20:24:39.038371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.038242 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7" event={"ID":"1b4351ba-b1cb-467d-ab0b-7158f7164dcc","Type":"ContainerDied","Data":"4f14bab9499cbdb222ce6857df3f0207d24a1df029d8964954b7c36e98e2d612"} Nov 03 20:24:39.038371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.038293 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f14bab9499cbdb222ce6857df3f0207d24a1df029d8964954b7c36e98e2d612" Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.111341 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9"] Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.112269 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1b4351ba-b1cb-467d-ab0b-7158f7164dcc" containerName="revoke-keys" Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.112289 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b4351ba-b1cb-467d-ab0b-7158f7164dcc" containerName="revoke-keys" Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.112354 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b27accbd-71f6-441a-bc62-9f4e1aa53f4b" containerName="revoke-keys" Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.112364 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b27accbd-71f6-441a-bc62-9f4e1aa53f4b" containerName="revoke-keys" Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.112542 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b4351ba-b1cb-467d-ab0b-7158f7164dcc" containerName="revoke-keys" Nov 03 20:24:39.114609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.112570 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b27accbd-71f6-441a-bc62-9f4e1aa53f4b" containerName="revoke-keys" Nov 03 20:24:39.116924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.116904 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.128134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.121050 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:24:39.128134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.122490 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:39.128134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.128079 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9"] Nov 03 20:24:39.192592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.192139 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.192592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.192336 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.192592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.192546 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxlz6\" (UniqueName: \"kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.235629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.235603 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:24:39.294007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.293970 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294212 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294258 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294299 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294297 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:39.294334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294327 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294396 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294443 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294555 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294588 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs\") pod \"922d5158-971d-45db-ad64-50fde71a4fba\" (UID: \"922d5158-971d-45db-ad64-50fde71a4fba\") " Nov 03 20:24:39.294860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.294835 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.295054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295034 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.295210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295164 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:39.295210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295189 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:39.295210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295201 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxlz6\" (UniqueName: \"kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.295466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295357 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.295466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295373 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.295466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295388 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922d5158-971d-45db-ad64-50fde71a4fba-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.295466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295449 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:39.296007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.295982 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:39.296158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.296131 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.298992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.298912 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.299390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.299345 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:39.299573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.299546 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "kube-api-access-fhc8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:39.300040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.300009 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:39.300706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.300682 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo" pod "922d5158-971d-45db-ad64-50fde71a4fba" (UID: "922d5158-971d-45db-ad64-50fde71a4fba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:39.308450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.308416 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxlz6\" (UniqueName: \"kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6\") pod \"maas-api-revoke-sub-5fd4b92f-q8ck9\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.421631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.420933 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.421631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.420971 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.421631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.420989 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922d5158-971d-45db-ad64-50fde71a4fba-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.421631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.421004 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhc8x\" (UniqueName: \"kubernetes.io/projected/922d5158-971d-45db-ad64-50fde71a4fba-kube-api-access-fhc8x\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.421631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.421017 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.421631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.421033 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922d5158-971d-45db-ad64-50fde71a4fba-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:39.454881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.454762 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:39.765648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:39.765622 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9"] Nov 03 20:24:40.054925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.054833 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" event={"ID":"922d5158-971d-45db-ad64-50fde71a4fba","Type":"ContainerDied","Data":"3e7a35c78aa0caa39f82ff412308e534ca06cd2a6ffd065d4dd34d1444209e93"} Nov 03 20:24:40.055404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.054951 5815 scope.go:122] "RemoveContainer" containerID="b5e4910b79ef44dd45e2fcbc4ed65e4b3a9549cee5a3684e85da5ba0848d5d1f" Nov 03 20:24:40.055825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.055584 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd" Nov 03 20:24:40.065929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.065899 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" event={"ID":"02e67380-5590-447a-a89b-02f673af0c8d","Type":"ContainerStarted","Data":"898f4e8075829fff98d03e0272624196426c4d9b08b7d325cb6d16e98046ecd1"} Nov 03 20:24:40.088197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.088167 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd"] Nov 03 20:24:40.091927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.091899 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9e5029-openshift-default-86bf88c645-s5lrd"] Nov 03 20:24:40.589778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:40.589741 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922d5158-971d-45db-ad64-50fde71a4fba" path="/var/lib/kubelet/pods/922d5158-971d-45db-ad64-50fde71a4fba/volumes" Nov 03 20:24:41.079421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:41.079389 5815 generic.go:356] "Generic (PLEG): container finished" podID="02e67380-5590-447a-a89b-02f673af0c8d" containerID="2aecf42386a1f3d6c9e3bbd32f8a29648b4dce5aa50889cdc9527b26a366b4fb" exitCode=0 Nov 03 20:24:41.079995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:41.079539 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" event={"ID":"02e67380-5590-447a-a89b-02f673af0c8d","Type":"ContainerDied","Data":"2aecf42386a1f3d6c9e3bbd32f8a29648b4dce5aa50889cdc9527b26a366b4fb"} Nov 03 20:24:42.267523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.267489 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:42.372950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.372871 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6\" (UniqueName: \"kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6\") pod \"02e67380-5590-447a-a89b-02f673af0c8d\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " Nov 03 20:24:42.373119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.372963 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca\") pod \"02e67380-5590-447a-a89b-02f673af0c8d\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " Nov 03 20:24:42.373188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.373159 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca\") pod \"02e67380-5590-447a-a89b-02f673af0c8d\" (UID: \"02e67380-5590-447a-a89b-02f673af0c8d\") " Nov 03 20:24:42.373348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.373317 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca" pod "02e67380-5590-447a-a89b-02f673af0c8d" (UID: "02e67380-5590-447a-a89b-02f673af0c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:42.373960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.373935 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:42.375283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.375259 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6" pod "02e67380-5590-447a-a89b-02f673af0c8d" (UID: "02e67380-5590-447a-a89b-02f673af0c8d"). InnerVolumeSpecName "kube-api-access-zxlz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:42.375584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.375561 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca" pod "02e67380-5590-447a-a89b-02f673af0c8d" (UID: "02e67380-5590-447a-a89b-02f673af0c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:42.479550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.479457 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxlz6\" (UniqueName: \"kubernetes.io/projected/02e67380-5590-447a-a89b-02f673af0c8d-kube-api-access-zxlz6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:42.479550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:42.479497 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e67380-5590-447a-a89b-02f673af0c8d-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:43.099106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:43.099063 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" event={"ID":"02e67380-5590-447a-a89b-02f673af0c8d","Type":"ContainerDied","Data":"898f4e8075829fff98d03e0272624196426c4d9b08b7d325cb6d16e98046ecd1"} Nov 03 20:24:43.099106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:43.099097 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9" Nov 03 20:24:43.099342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:43.099122 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898f4e8075829fff98d03e0272624196426c4d9b08b7d325cb6d16e98046ecd1" Nov 03 20:24:43.915310 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:24:43.915268 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:24:44.143959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.143930 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd"] Nov 03 20:24:44.144604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.144589 5815 cpu_manager.go:414] "containerMap: removing container" podUID="02e67380-5590-447a-a89b-02f673af0c8d" containerName="revoke-keys" Nov 03 20:24:44.144668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.144606 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e67380-5590-447a-a89b-02f673af0c8d" containerName="revoke-keys" Nov 03 20:24:44.144668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.144636 5815 cpu_manager.go:414] "containerMap: removing container" podUID="922d5158-971d-45db-ad64-50fde71a4fba" containerName="istio-proxy" Nov 03 20:24:44.144668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.144641 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922d5158-971d-45db-ad64-50fde71a4fba" containerName="istio-proxy" Nov 03 20:24:44.144760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.144743 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e67380-5590-447a-a89b-02f673af0c8d" containerName="revoke-keys" Nov 03 20:24:44.144760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.144758 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="922d5158-971d-45db-ad64-50fde71a4fba" containerName="istio-proxy" Nov 03 20:24:44.148309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.148294 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.154387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.154356 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:44.154613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.154446 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:24:44.172583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.158447 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd"] Nov 03 20:24:44.317257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.317222 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6qf\" (UniqueName: \"kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.317445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.317419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.317717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.317693 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.419402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.419359 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.419587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.419490 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.419645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.419600 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6qf\" (UniqueName: \"kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.420041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.420017 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.421982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.421963 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.431944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.431886 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6qf\" (UniqueName: \"kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf\") pod \"maas-api-revoke-sub-63693ee7-7gmnd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.466725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.466699 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:44.757713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:44.757687 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd"] Nov 03 20:24:44.758318 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:24:44.758292 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813a84c2_5db4_4537_b612_c14de26429bd.slice/crio-7d01093aaa2fceb7f2f2cf864187116e54ae5df7af4e5377ea715af94d360059 WatchSource:0}: Error finding container 7d01093aaa2fceb7f2f2cf864187116e54ae5df7af4e5377ea715af94d360059: Status 404 returned error can't find the container with id 7d01093aaa2fceb7f2f2cf864187116e54ae5df7af4e5377ea715af94d360059 Nov 03 20:24:45.119777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:45.119745 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" event={"ID":"813a84c2-5db4-4537-b612-c14de26429bd","Type":"ContainerStarted","Data":"49f3ccd181782709ce9491cac727e8a3bd9f3d7dd49fdb9403f602c0a6b599f9"} Nov 03 20:24:45.120184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:45.119799 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" event={"ID":"813a84c2-5db4-4537-b612-c14de26429bd","Type":"ContainerStarted","Data":"7d01093aaa2fceb7f2f2cf864187116e54ae5df7af4e5377ea715af94d360059"} Nov 03 20:24:45.140401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:45.140358 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" podStartSLOduration=1.140344627 podStartE2EDuration="1.140344627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:45.13690512 +0000 UTC m=+1394.667250794" watchObservedRunningTime="2026-11-03 20:24:45.140344627 +0000 UTC m=+1394.670690312" Nov 03 20:24:45.243400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:45.243372 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:24:46.133519 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:46.133402 5815 generic.go:356] "Generic (PLEG): container finished" podID="813a84c2-5db4-4537-b612-c14de26429bd" containerID="49f3ccd181782709ce9491cac727e8a3bd9f3d7dd49fdb9403f602c0a6b599f9" exitCode=0 Nov 03 20:24:46.133992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:46.133564 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" event={"ID":"813a84c2-5db4-4537-b612-c14de26429bd","Type":"ContainerDied","Data":"49f3ccd181782709ce9491cac727e8a3bd9f3d7dd49fdb9403f602c0a6b599f9"} Nov 03 20:24:47.130040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.130004 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc"] Nov 03 20:24:47.281542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.267561 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc"] Nov 03 20:24:47.283706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.283684 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.514546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.501200 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.514546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.501360 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxdw\" (UniqueName: \"kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.514546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.501431 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.644533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.615747 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.644533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.641806 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxdw\" (UniqueName: \"kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.644533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.641901 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.644533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.643677 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.697552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.686489 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxdw\" (UniqueName: \"kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.735064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.735001 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de05a226-wkdpc\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:47.822613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.810195 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5"] Nov 03 20:24:48.058535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:47.996604 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv"] Nov 03 20:24:48.058535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.011058 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:48.206861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.206755 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.422364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.422316 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.488531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.482835 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.508869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.508827 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.595319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.595254 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.832582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.811570 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:24:48.846591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.846555 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk"] Nov 03 20:24:48.849710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:48.849658 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb77820-l5rrk"] Nov 03 20:24:49.470549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.455035 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:49.470549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.455420 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:24:49.470549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.464176 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:24:49.470549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.464247 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:24:49.584971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.584679 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:49.668718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.667676 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:24:49.748201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.745541 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb"] Nov 03 20:24:49.748201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.746462 5815 cpu_manager.go:414] "containerMap: removing container" podUID="813a84c2-5db4-4537-b612-c14de26429bd" containerName="revoke-keys" Nov 03 20:24:49.748201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.746478 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813a84c2-5db4-4537-b612-c14de26429bd" containerName="revoke-keys" Nov 03 20:24:49.748201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.746734 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="813a84c2-5db4-4537-b612-c14de26429bd" containerName="revoke-keys" Nov 03 20:24:49.777680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.770126 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf\" (UniqueName: \"kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf\") pod \"813a84c2-5db4-4537-b612-c14de26429bd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " Nov 03 20:24:49.777680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.770243 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca\") pod \"813a84c2-5db4-4537-b612-c14de26429bd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " Nov 03 20:24:49.777680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.770318 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca\") pod \"813a84c2-5db4-4537-b612-c14de26429bd\" (UID: \"813a84c2-5db4-4537-b612-c14de26429bd\") " Nov 03 20:24:49.779141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.779098 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca" pod "813a84c2-5db4-4537-b612-c14de26429bd" (UID: "813a84c2-5db4-4537-b612-c14de26429bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:49.811424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.807171 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb"] Nov 03 20:24:49.811424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.807222 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc"] Nov 03 20:24:49.811424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.807309 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:24:49.811424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.807398 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:49.840838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.840340 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" event={"ID":"813a84c2-5db4-4537-b612-c14de26429bd","Type":"ContainerDied","Data":"7d01093aaa2fceb7f2f2cf864187116e54ae5df7af4e5377ea715af94d360059"} Nov 03 20:24:49.840838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.840393 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d01093aaa2fceb7f2f2cf864187116e54ae5df7af4e5377ea715af94d360059" Nov 03 20:24:49.840838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.840454 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd" Nov 03 20:24:49.852592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.844267 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf" pod "813a84c2-5db4-4537-b612-c14de26429bd" (UID: "813a84c2-5db4-4537-b612-c14de26429bd"). InnerVolumeSpecName "kube-api-access-jq6qf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:49.859553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.856667 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca" pod "813a84c2-5db4-4537-b612-c14de26429bd" (UID: "813a84c2-5db4-4537-b612-c14de26429bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:49.877703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.875371 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6qf\" (UniqueName: \"kubernetes.io/projected/813a84c2-5db4-4537-b612-c14de26429bd-kube-api-access-jq6qf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:49.877703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.875399 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:49.877703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.875414 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/813a84c2-5db4-4537-b612-c14de26429bd-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:49.977261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.977224 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxlx\" (UniqueName: \"kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:49.977438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.977373 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:49.977664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:49.977638 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.079693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.079610 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxlx\" (UniqueName: \"kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.079840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.079696 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.079840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.079790 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.080580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.080547 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.083094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.083068 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.092294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.092272 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxlx\" (UniqueName: \"kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx\") pod \"maas-api-revoke-sub-793d11d7-f6nkb\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.187957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.187930 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:50.552621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.552577 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb"] Nov 03 20:24:50.553147 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:24:50.552875 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7dad49_c112_4215_95ba_9fbfe5d911c0.slice/crio-21a1c7d415f4e9878c835754ff35eac46381127207b7a47620fb51397a5b43db WatchSource:0}: Error finding container 21a1c7d415f4e9878c835754ff35eac46381127207b7a47620fb51397a5b43db: Status 404 returned error can't find the container with id 21a1c7d415f4e9878c835754ff35eac46381127207b7a47620fb51397a5b43db Nov 03 20:24:50.615558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.615485 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3" path="/var/lib/kubelet/pods/24d36ffc-7ba6-4979-8b2c-bd705c2ea6e3/volumes" Nov 03 20:24:50.851048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.850961 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" event={"ID":"14cf1080-ad76-4f85-8cba-033b0e254267","Type":"ContainerStarted","Data":"2c4fd6bd873e2aa2640d1bc82fa7a41311355035adbdcbde00063ac4e7d89fa1"} Nov 03 20:24:50.851048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.851029 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" event={"ID":"14cf1080-ad76-4f85-8cba-033b0e254267","Type":"ContainerStarted","Data":"669d1b4bbe6f4f1fa858a0dab8ca774f6e3a050bc622773d94e5b8bbad3a2845"} Nov 03 20:24:50.852541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.852498 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" event={"ID":"ab7dad49-c112-4215-95ba-9fbfe5d911c0","Type":"ContainerStarted","Data":"21a1c7d415f4e9878c835754ff35eac46381127207b7a47620fb51397a5b43db"} Nov 03 20:24:50.869850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:50.869782 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" podStartSLOduration=3.869766028 podStartE2EDuration="3.869766028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:50.868794334 +0000 UTC m=+1400.399140007" watchObservedRunningTime="2026-11-03 20:24:50.869766028 +0000 UTC m=+1400.400111702" Nov 03 20:24:51.862406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:51.862375 5815 generic.go:356] "Generic (PLEG): container finished" podID="14cf1080-ad76-4f85-8cba-033b0e254267" containerID="2c4fd6bd873e2aa2640d1bc82fa7a41311355035adbdcbde00063ac4e7d89fa1" exitCode=0 Nov 03 20:24:51.862823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:51.862453 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" event={"ID":"14cf1080-ad76-4f85-8cba-033b0e254267","Type":"ContainerDied","Data":"2c4fd6bd873e2aa2640d1bc82fa7a41311355035adbdcbde00063ac4e7d89fa1"} Nov 03 20:24:51.864402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:51.864375 5815 generic.go:356] "Generic (PLEG): container finished" podID="ab7dad49-c112-4215-95ba-9fbfe5d911c0" containerID="9a49aa17f891a944edcce0ed1f2f8342b0ea4bf1877ea884dc547650e80dc4f6" exitCode=0 Nov 03 20:24:51.864578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:51.864441 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" event={"ID":"ab7dad49-c112-4215-95ba-9fbfe5d911c0","Type":"ContainerDied","Data":"9a49aa17f891a944edcce0ed1f2f8342b0ea4bf1877ea884dc547650e80dc4f6"} Nov 03 20:24:52.815664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:52.812583 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" podUID="f788d31a-b296-4f14-8f1d-37c12df46426" containerName="payload-processing" containerID="cri-o://cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81" gracePeriod=25 Nov 03 20:24:52.998994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:52.998881 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" podUID="a7ad037d-f81b-4878-b0e5-81c5db3dfbff" containerName="payload-pre-processing" containerID="cri-o://45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0" gracePeriod=25 Nov 03 20:24:53.146123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.146101 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:53.149549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.149527 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:53.164390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.164362 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:24:53.300952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.300255 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5\" (UniqueName: \"kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5\") pod \"f788d31a-b296-4f14-8f1d-37c12df46426\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " Nov 03 20:24:53.300952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.300354 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca\") pod \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " Nov 03 20:24:53.300952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.300394 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca\") pod \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " Nov 03 20:24:53.300952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.300430 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca\") pod \"14cf1080-ad76-4f85-8cba-033b0e254267\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " Nov 03 20:24:53.300952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.300669 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca\") pod \"14cf1080-ad76-4f85-8cba-033b0e254267\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " Nov 03 20:24:53.300952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.300732 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume\") pod \"f788d31a-b296-4f14-8f1d-37c12df46426\" (UID: \"f788d31a-b296-4f14-8f1d-37c12df46426\") " Nov 03 20:24:53.301433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.301149 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx\" (UniqueName: \"kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx\") pod \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\" (UID: \"ab7dad49-c112-4215-95ba-9fbfe5d911c0\") " Nov 03 20:24:53.301433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.301226 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw\" (UniqueName: \"kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw\") pod \"14cf1080-ad76-4f85-8cba-033b0e254267\" (UID: \"14cf1080-ad76-4f85-8cba-033b0e254267\") " Nov 03 20:24:53.304143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.303519 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume" pod "f788d31a-b296-4f14-8f1d-37c12df46426" (UID: "f788d31a-b296-4f14-8f1d-37c12df46426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:53.304143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.303689 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca" pod "ab7dad49-c112-4215-95ba-9fbfe5d911c0" (UID: "ab7dad49-c112-4215-95ba-9fbfe5d911c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:53.304143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.303920 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca" pod "14cf1080-ad76-4f85-8cba-033b0e254267" (UID: "14cf1080-ad76-4f85-8cba-033b0e254267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:53.323056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.322994 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx" pod "ab7dad49-c112-4215-95ba-9fbfe5d911c0" (UID: "ab7dad49-c112-4215-95ba-9fbfe5d911c0"). InnerVolumeSpecName "kube-api-access-xmxlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:53.323808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.323759 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca" pod "ab7dad49-c112-4215-95ba-9fbfe5d911c0" (UID: "ab7dad49-c112-4215-95ba-9fbfe5d911c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:53.324023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.323999 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca" pod "14cf1080-ad76-4f85-8cba-033b0e254267" (UID: "14cf1080-ad76-4f85-8cba-033b0e254267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:53.324300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.324287 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw" pod "14cf1080-ad76-4f85-8cba-033b0e254267" (UID: "14cf1080-ad76-4f85-8cba-033b0e254267"). InnerVolumeSpecName "kube-api-access-jmxdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:53.324576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.324549 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5" pod "f788d31a-b296-4f14-8f1d-37c12df46426" (UID: "f788d31a-b296-4f14-8f1d-37c12df46426"). InnerVolumeSpecName "kube-api-access-ftxc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:53.335450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.335426 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:24:53.402766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.402674 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume\") pod \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " Nov 03 20:24:53.403031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.403010 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz\" (UniqueName: \"kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz\") pod \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\" (UID: \"a7ad037d-f81b-4878-b0e5-81c5db3dfbff\") " Nov 03 20:24:53.404031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.403943 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxlx\" (UniqueName: \"kubernetes.io/projected/ab7dad49-c112-4215-95ba-9fbfe5d911c0-kube-api-access-xmxlx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404041 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxdw\" (UniqueName: \"kubernetes.io/projected/14cf1080-ad76-4f85-8cba-033b0e254267-kube-api-access-jmxdw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404058 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxc5\" (UniqueName: \"kubernetes.io/projected/f788d31a-b296-4f14-8f1d-37c12df46426-kube-api-access-ftxc5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404089 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404106 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7dad49-c112-4215-95ba-9fbfe5d911c0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404122 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404136 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14cf1080-ad76-4f85-8cba-033b0e254267-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404164 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f788d31a-b296-4f14-8f1d-37c12df46426-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.404964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.404929 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume" pod "a7ad037d-f81b-4878-b0e5-81c5db3dfbff" (UID: "a7ad037d-f81b-4878-b0e5-81c5db3dfbff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:53.409263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.409236 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz" pod "a7ad037d-f81b-4878-b0e5-81c5db3dfbff" (UID: "a7ad037d-f81b-4878-b0e5-81c5db3dfbff"). InnerVolumeSpecName "kube-api-access-55pxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:53.524639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.524607 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pxz\" (UniqueName: \"kubernetes.io/projected/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-kube-api-access-55pxz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.524639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.524640 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ad037d-f81b-4878-b0e5-81c5db3dfbff-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.885058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.885024 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" Nov 03 20:24:53.885238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.885066 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb" event={"ID":"ab7dad49-c112-4215-95ba-9fbfe5d911c0","Type":"ContainerDied","Data":"21a1c7d415f4e9878c835754ff35eac46381127207b7a47620fb51397a5b43db"} Nov 03 20:24:53.885238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.885123 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21a1c7d415f4e9878c835754ff35eac46381127207b7a47620fb51397a5b43db" Nov 03 20:24:53.888126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.888105 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" Nov 03 20:24:53.888286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.888106 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc" event={"ID":"14cf1080-ad76-4f85-8cba-033b0e254267","Type":"ContainerDied","Data":"669d1b4bbe6f4f1fa858a0dab8ca774f6e3a050bc622773d94e5b8bbad3a2845"} Nov 03 20:24:53.888286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.888231 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="669d1b4bbe6f4f1fa858a0dab8ca774f6e3a050bc622773d94e5b8bbad3a2845" Nov 03 20:24:53.889875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.889851 5815 generic.go:356] "Generic (PLEG): container finished" podID="a7ad037d-f81b-4878-b0e5-81c5db3dfbff" containerID="45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0" exitCode=0 Nov 03 20:24:53.890008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.889888 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" Nov 03 20:24:53.890008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.889917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" event={"ID":"a7ad037d-f81b-4878-b0e5-81c5db3dfbff","Type":"ContainerDied","Data":"45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0"} Nov 03 20:24:53.890008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.889969 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv" event={"ID":"a7ad037d-f81b-4878-b0e5-81c5db3dfbff","Type":"ContainerDied","Data":"525c5c0af0f7ff9f11a52ad43601450a059cb1f9f226cdb8b25a8a17cad8f420"} Nov 03 20:24:53.890008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.889987 5815 scope.go:122] "RemoveContainer" containerID="45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0" Nov 03 20:24:53.895690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.892620 5815 generic.go:356] "Generic (PLEG): container finished" podID="f788d31a-b296-4f14-8f1d-37c12df46426" containerID="cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81" exitCode=0 Nov 03 20:24:53.895690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.892679 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" Nov 03 20:24:53.895690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.892678 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" event={"ID":"f788d31a-b296-4f14-8f1d-37c12df46426","Type":"ContainerDied","Data":"cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81"} Nov 03 20:24:53.895690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.894241 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5" event={"ID":"f788d31a-b296-4f14-8f1d-37c12df46426","Type":"ContainerDied","Data":"ef28c229f616b96d8ed26e1c7b50cae7228284116530ca704f6cafaacc4ac474"} Nov 03 20:24:53.903746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.903702 5815 scope.go:122] "RemoveContainer" containerID="45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0" Nov 03 20:24:53.904114 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:24:53.904076 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0\": container with ID starting with 45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0 not found: ID does not exist" containerID="45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0" Nov 03 20:24:53.904241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.904111 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0"} err="failed to get container status \"45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0\": rpc error: code = NotFound desc = could not find container \"45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0\": container with ID starting with 45de3630f2a5cfbf6a6699b1b2cb6c48a8edb92e8fd7dc01eb7acd317423c8f0 not found: ID does not exist" Nov 03 20:24:53.904241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.904133 5815 scope.go:122] "RemoveContainer" containerID="cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81" Nov 03 20:24:53.916153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.916136 5815 scope.go:122] "RemoveContainer" containerID="cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81" Nov 03 20:24:53.916442 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:24:53.916416 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81\": container with ID starting with cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81 not found: ID does not exist" containerID="cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81" Nov 03 20:24:53.916588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.916445 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81"} err="failed to get container status \"cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81\": rpc error: code = NotFound desc = could not find container \"cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81\": container with ID starting with cda6b0ffa8a6063f8ca546c9745133bdc8dcf11e603effc129a0644bd0587f81 not found: ID does not exist" Nov 03 20:24:53.927672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.927654 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5"] Nov 03 20:24:53.932314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.932291 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-019ef4-68fc7d79d-h48b5"] Nov 03 20:24:53.979699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:53.979654 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv"] Nov 03 20:24:54.011246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.000456 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-019ef4-88cccb99f-grxwv"] Nov 03 20:24:54.456305 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456273 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65"] Nov 03 20:24:54.456870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456857 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ab7dad49-c112-4215-95ba-9fbfe5d911c0" containerName="revoke-keys" Nov 03 20:24:54.456933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456871 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7dad49-c112-4215-95ba-9fbfe5d911c0" containerName="revoke-keys" Nov 03 20:24:54.456933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456901 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a7ad037d-f81b-4878-b0e5-81c5db3dfbff" containerName="payload-pre-processing" Nov 03 20:24:54.456933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456906 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ad037d-f81b-4878-b0e5-81c5db3dfbff" containerName="payload-pre-processing" Nov 03 20:24:54.456933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456920 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f788d31a-b296-4f14-8f1d-37c12df46426" containerName="payload-processing" Nov 03 20:24:54.456933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456925 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f788d31a-b296-4f14-8f1d-37c12df46426" containerName="payload-processing" Nov 03 20:24:54.456933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456933 5815 cpu_manager.go:414] "containerMap: removing container" podUID="14cf1080-ad76-4f85-8cba-033b0e254267" containerName="revoke-keys" Nov 03 20:24:54.457116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.456938 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cf1080-ad76-4f85-8cba-033b0e254267" containerName="revoke-keys" Nov 03 20:24:54.457116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.457061 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ad037d-f81b-4878-b0e5-81c5db3dfbff" containerName="payload-pre-processing" Nov 03 20:24:54.457116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.457068 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f788d31a-b296-4f14-8f1d-37c12df46426" containerName="payload-processing" Nov 03 20:24:54.457116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.457075 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cf1080-ad76-4f85-8cba-033b0e254267" containerName="revoke-keys" Nov 03 20:24:54.457116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.457086 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7dad49-c112-4215-95ba-9fbfe5d911c0" containerName="revoke-keys" Nov 03 20:24:54.461791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.461774 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.469399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.469375 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65"] Nov 03 20:24:54.482296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.481996 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:24:54.482296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.482058 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:54.550274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.550244 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.550435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.550280 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkq7t\" (UniqueName: \"kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.550549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.550446 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.590781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.590749 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ad037d-f81b-4878-b0e5-81c5db3dfbff" path="/var/lib/kubelet/pods/a7ad037d-f81b-4878-b0e5-81c5db3dfbff/volumes" Nov 03 20:24:54.591195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.591182 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f788d31a-b296-4f14-8f1d-37c12df46426" path="/var/lib/kubelet/pods/f788d31a-b296-4f14-8f1d-37c12df46426/volumes" Nov 03 20:24:54.652210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.652183 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.652373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.652278 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.652373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.652298 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkq7t\" (UniqueName: \"kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.652929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.652909 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.654735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.654712 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.664280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.664254 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkq7t\" (UniqueName: \"kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t\") pod \"maas-api-revoke-sub-5d89d48f-76r65\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:54.788545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:54.788500 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:55.074584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:55.074552 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65"] Nov 03 20:24:55.074978 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:24:55.074907 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode465645b_7a0f_47b4_b52e_ff005b316b1b.slice/crio-135fcab0a78e95a04680f6253dc7e6497ca4389b44d018824eb771c75243ee92 WatchSource:0}: Error finding container 135fcab0a78e95a04680f6253dc7e6497ca4389b44d018824eb771c75243ee92: Status 404 returned error can't find the container with id 135fcab0a78e95a04680f6253dc7e6497ca4389b44d018824eb771c75243ee92 Nov 03 20:24:55.912607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:55.907570 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59"] Nov 03 20:24:55.912607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:55.907836 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" containerID="cri-o://575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed" gracePeriod=30 Nov 03 20:24:56.368578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:56.367752 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" event={"ID":"e465645b-7a0f-47b4-b52e-ff005b316b1b","Type":"ContainerStarted","Data":"4548dfe795e575b3e6d5db0ad6ec9caf851a88fc9703d03b84c83e9a4992c39a"} Nov 03 20:24:56.368578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:56.367819 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" event={"ID":"e465645b-7a0f-47b4-b52e-ff005b316b1b","Type":"ContainerStarted","Data":"135fcab0a78e95a04680f6253dc7e6497ca4389b44d018824eb771c75243ee92"} Nov 03 20:24:56.400600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:56.400548 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" podStartSLOduration=2.400527583 podStartE2EDuration="2.400527583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:56.393904271 +0000 UTC m=+1405.924249950" watchObservedRunningTime="2026-11-03 20:24:56.400527583 +0000 UTC m=+1405.930873256" Nov 03 20:24:57.377312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:57.377276 5815 generic.go:356] "Generic (PLEG): container finished" podID="e465645b-7a0f-47b4-b52e-ff005b316b1b" containerID="4548dfe795e575b3e6d5db0ad6ec9caf851a88fc9703d03b84c83e9a4992c39a" exitCode=0 Nov 03 20:24:57.377794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:57.377366 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" event={"ID":"e465645b-7a0f-47b4-b52e-ff005b316b1b","Type":"ContainerDied","Data":"4548dfe795e575b3e6d5db0ad6ec9caf851a88fc9703d03b84c83e9a4992c39a"} Nov 03 20:24:58.627694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.627667 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:58.716140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.716100 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca\") pod \"e465645b-7a0f-47b4-b52e-ff005b316b1b\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " Nov 03 20:24:58.716314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.716214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t\" (UniqueName: \"kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t\") pod \"e465645b-7a0f-47b4-b52e-ff005b316b1b\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " Nov 03 20:24:58.716395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.716344 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca\") pod \"e465645b-7a0f-47b4-b52e-ff005b316b1b\" (UID: \"e465645b-7a0f-47b4-b52e-ff005b316b1b\") " Nov 03 20:24:58.716890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.716852 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca" pod "e465645b-7a0f-47b4-b52e-ff005b316b1b" (UID: "e465645b-7a0f-47b4-b52e-ff005b316b1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:58.717636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.717433 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:58.718903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.718866 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca" pod "e465645b-7a0f-47b4-b52e-ff005b316b1b" (UID: "e465645b-7a0f-47b4-b52e-ff005b316b1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:58.719339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.719313 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t" pod "e465645b-7a0f-47b4-b52e-ff005b316b1b" (UID: "e465645b-7a0f-47b4-b52e-ff005b316b1b"). InnerVolumeSpecName "kube-api-access-kkq7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:58.819287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.819256 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e465645b-7a0f-47b4-b52e-ff005b316b1b-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:58.819287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:58.819287 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkq7t\" (UniqueName: \"kubernetes.io/projected/e465645b-7a0f-47b4-b52e-ff005b316b1b-kube-api-access-kkq7t\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:59.397701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:59.397671 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" Nov 03 20:24:59.397701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:59.397687 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65" event={"ID":"e465645b-7a0f-47b4-b52e-ff005b316b1b","Type":"ContainerDied","Data":"135fcab0a78e95a04680f6253dc7e6497ca4389b44d018824eb771c75243ee92"} Nov 03 20:24:59.397906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:24:59.397759 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="135fcab0a78e95a04680f6253dc7e6497ca4389b44d018824eb771c75243ee92" Nov 03 20:25:01.735697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.735641 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:25:01.894301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894266 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894307 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894390 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894421 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894539 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894588 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894633 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894702 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.894836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894789 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:01.894836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894808 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:01.894974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894800 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data\") pod \"66e98e0c-1154-4ad3-b860-78439f986d28\" (UID: \"66e98e0c-1154-4ad3-b860-78439f986d28\") " Nov 03 20:25:01.895052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.894971 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:01.895407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895383 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:01.895627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895499 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:01.895887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895860 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e98e0c-1154-4ad3-b860-78439f986d28-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.895992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895889 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.895992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895904 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.895992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895917 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.895992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.895931 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.897013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.896988 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "kube-api-access-8m86c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:01.897093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.897032 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:01.897137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.897091 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:01.897681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.897659 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy" pod "66e98e0c-1154-4ad3-b860-78439f986d28" (UID: "66e98e0c-1154-4ad3-b860-78439f986d28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:01.996843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.996765 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e98e0c-1154-4ad3-b860-78439f986d28-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.996843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.996792 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m86c\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-kube-api-access-8m86c\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.996843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.996805 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e98e0c-1154-4ad3-b860-78439f986d28-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:01.996843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:01.996817 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e98e0c-1154-4ad3-b860-78439f986d28-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:02.163852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.163807 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj"] Nov 03 20:25:02.164480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164461 5815 cpu_manager.go:414] "containerMap: removing container" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" Nov 03 20:25:02.164624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164482 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" Nov 03 20:25:02.164624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164538 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e465645b-7a0f-47b4-b52e-ff005b316b1b" containerName="revoke-keys" Nov 03 20:25:02.164624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164547 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e465645b-7a0f-47b4-b52e-ff005b316b1b" containerName="revoke-keys" Nov 03 20:25:02.164792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164735 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" Nov 03 20:25:02.164792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164753 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e465645b-7a0f-47b4-b52e-ff005b316b1b" containerName="revoke-keys" Nov 03 20:25:02.164792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.164771 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" Nov 03 20:25:02.168558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.168539 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.172930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.172892 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:02.173057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.173031 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:25:02.180490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.180468 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj"] Nov 03 20:25:02.300669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.300580 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.300669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.300623 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.300904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.300807 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rql\" (UniqueName: \"kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.401948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.401921 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.401948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.401952 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.402167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.402055 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rql\" (UniqueName: \"kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.402678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.402652 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.404466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.404441 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.414232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.414209 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rql\" (UniqueName: \"kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql\") pod \"maas-api-revoke-sub-98bd36cc-pjqjj\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.486928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.486898 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:02.499139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.499104 5815 generic.go:356] "Generic (PLEG): container finished" podID="66e98e0c-1154-4ad3-b860-78439f986d28" containerID="575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed" exitCode=0 Nov 03 20:25:02.499281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.499171 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" Nov 03 20:25:02.499281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.499191 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" event={"ID":"66e98e0c-1154-4ad3-b860-78439f986d28","Type":"ContainerDied","Data":"575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed"} Nov 03 20:25:02.499413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.499286 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59" event={"ID":"66e98e0c-1154-4ad3-b860-78439f986d28","Type":"ContainerDied","Data":"58032a486c671358c7aad4f3a90b39e3841e9af67c63fb2c2c8c290c3cc21e93"} Nov 03 20:25:02.499413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.499318 5815 scope.go:122] "RemoveContainer" containerID="575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed" Nov 03 20:25:02.548539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.544759 5815 scope.go:122] "RemoveContainer" containerID="3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60" Nov 03 20:25:02.650912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.650838 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59"] Nov 03 20:25:02.650912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.650871 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-019ef4-openshift-default-768d646ffc-5ch59"] Nov 03 20:25:02.657677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.657594 5815 scope.go:122] "RemoveContainer" containerID="575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed" Nov 03 20:25:02.658174 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:25:02.657908 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed\": container with ID starting with 575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed not found: ID does not exist" containerID="575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed" Nov 03 20:25:02.658174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.657938 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed"} err="failed to get container status \"575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed\": rpc error: code = NotFound desc = could not find container \"575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed\": container with ID starting with 575c69fecbf465cc25d1450894e66439fee3bf30436c71aeaa9e249d5d7568ed not found: ID does not exist" Nov 03 20:25:02.658174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.657956 5815 scope.go:122] "RemoveContainer" containerID="3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60" Nov 03 20:25:02.658419 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:25:02.658211 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60\": container with ID starting with 3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60 not found: ID does not exist" containerID="3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60" Nov 03 20:25:02.658419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.658250 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60"} err="failed to get container status \"3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60\": rpc error: code = NotFound desc = could not find container \"3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60\": container with ID starting with 3e3bb0a79e8f2975d4ea2ead45b31cac7f6d88dd89f1b1dec20d53f3f599fe60 not found: ID does not exist" Nov 03 20:25:02.830518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:02.830481 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj"] Nov 03 20:25:02.831448 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:02.831421 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23e8318_e623_4dd1_ba53_87eac194fbcf.slice/crio-b7418a6ad6dc2aca9ee08ac2453cdc670f16f930f72299a725f1ea6655c3c9a6 WatchSource:0}: Error finding container b7418a6ad6dc2aca9ee08ac2453cdc670f16f930f72299a725f1ea6655c3c9a6: Status 404 returned error can't find the container with id b7418a6ad6dc2aca9ee08ac2453cdc670f16f930f72299a725f1ea6655c3c9a6 Nov 03 20:25:03.471732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.471704 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc"] Nov 03 20:25:03.472304 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.472289 5815 cpu_manager.go:414] "containerMap: removing container" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" Nov 03 20:25:03.472369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.472305 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" containerName="istio-proxy" Nov 03 20:25:03.476114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.476093 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.484781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.484758 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc"] Nov 03 20:25:03.515424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.515393 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" event={"ID":"a23e8318-e623-4dd1-ba53-87eac194fbcf","Type":"ContainerStarted","Data":"9558adf2b457c2ad9928994356158c5a401394682bcf47f4d3cd43dd176ea522"} Nov 03 20:25:03.515633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.515451 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" event={"ID":"a23e8318-e623-4dd1-ba53-87eac194fbcf","Type":"ContainerStarted","Data":"b7418a6ad6dc2aca9ee08ac2453cdc670f16f930f72299a725f1ea6655c3c9a6"} Nov 03 20:25:03.537429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.537385 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" podStartSLOduration=1.5373731400000001 podStartE2EDuration="1.53737314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:03.533021145 +0000 UTC m=+1413.063366820" watchObservedRunningTime="2026-11-03 20:25:03.53737314 +0000 UTC m=+1413.067718812" Nov 03 20:25:03.556833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.556805 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnskj\" (UniqueName: \"kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.556980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.556948 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.557070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.557031 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.658584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.658545 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.658730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.658656 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.658829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.658812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnskj\" (UniqueName: \"kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.659029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.659007 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.661190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.661168 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.671446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.671416 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnskj\" (UniqueName: \"kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj\") pod \"maas-api-revoke-sub-b378a5a0-g8xpc\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:03.803535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:03.793009 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:04.115163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.115139 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc"] Nov 03 20:25:04.116120 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:04.116090 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fb2a69_834a_47fd_91dd_95f09b5b4c5f.slice/crio-9a4c8573e9bc9112e4e36bc5b0e0236da6541906e67bdbd777730addb107b677 WatchSource:0}: Error finding container 9a4c8573e9bc9112e4e36bc5b0e0236da6541906e67bdbd777730addb107b677: Status 404 returned error can't find the container with id 9a4c8573e9bc9112e4e36bc5b0e0236da6541906e67bdbd777730addb107b677 Nov 03 20:25:04.527212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.527166 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" event={"ID":"40fb2a69-834a-47fd-91dd-95f09b5b4c5f","Type":"ContainerStarted","Data":"845d4288ec8f2a7986b207527167e18f8c14017dae73c68b2dead9057de5c523"} Nov 03 20:25:04.527385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.527222 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" event={"ID":"40fb2a69-834a-47fd-91dd-95f09b5b4c5f","Type":"ContainerStarted","Data":"9a4c8573e9bc9112e4e36bc5b0e0236da6541906e67bdbd777730addb107b677"} Nov 03 20:25:04.529542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.529473 5815 generic.go:356] "Generic (PLEG): container finished" podID="a23e8318-e623-4dd1-ba53-87eac194fbcf" containerID="9558adf2b457c2ad9928994356158c5a401394682bcf47f4d3cd43dd176ea522" exitCode=0 Nov 03 20:25:04.529673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.529555 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" event={"ID":"a23e8318-e623-4dd1-ba53-87eac194fbcf","Type":"ContainerDied","Data":"9558adf2b457c2ad9928994356158c5a401394682bcf47f4d3cd43dd176ea522"} Nov 03 20:25:04.549677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.549631 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" podStartSLOduration=1.549619067 podStartE2EDuration="1.549619067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:04.548099501 +0000 UTC m=+1414.078445175" watchObservedRunningTime="2026-11-03 20:25:04.549619067 +0000 UTC m=+1414.079964736" Nov 03 20:25:04.589259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:04.589226 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e98e0c-1154-4ad3-b860-78439f986d28" path="/var/lib/kubelet/pods/66e98e0c-1154-4ad3-b860-78439f986d28/volumes" Nov 03 20:25:05.539435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.539400 5815 generic.go:356] "Generic (PLEG): container finished" podID="40fb2a69-834a-47fd-91dd-95f09b5b4c5f" containerID="845d4288ec8f2a7986b207527167e18f8c14017dae73c68b2dead9057de5c523" exitCode=0 Nov 03 20:25:05.539877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.539488 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" event={"ID":"40fb2a69-834a-47fd-91dd-95f09b5b4c5f","Type":"ContainerDied","Data":"845d4288ec8f2a7986b207527167e18f8c14017dae73c68b2dead9057de5c523"} Nov 03 20:25:05.733568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.733543 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:05.805596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.800648 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca\") pod \"a23e8318-e623-4dd1-ba53-87eac194fbcf\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " Nov 03 20:25:05.805596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.800711 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql\" (UniqueName: \"kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql\") pod \"a23e8318-e623-4dd1-ba53-87eac194fbcf\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " Nov 03 20:25:05.805596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.804703 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql" pod "a23e8318-e623-4dd1-ba53-87eac194fbcf" (UID: "a23e8318-e623-4dd1-ba53-87eac194fbcf"). InnerVolumeSpecName "kube-api-access-49rql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:05.826739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.825654 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca\") pod \"a23e8318-e623-4dd1-ba53-87eac194fbcf\" (UID: \"a23e8318-e623-4dd1-ba53-87eac194fbcf\") " Nov 03 20:25:05.826739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.826417 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rql\" (UniqueName: \"kubernetes.io/projected/a23e8318-e623-4dd1-ba53-87eac194fbcf-kube-api-access-49rql\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:05.826966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.826798 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca" pod "a23e8318-e623-4dd1-ba53-87eac194fbcf" (UID: "a23e8318-e623-4dd1-ba53-87eac194fbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:05.837540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.828143 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca" pod "a23e8318-e623-4dd1-ba53-87eac194fbcf" (UID: "a23e8318-e623-4dd1-ba53-87eac194fbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:05.928361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.928328 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:05.928361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:05.928357 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a23e8318-e623-4dd1-ba53-87eac194fbcf-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:06.548826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.548798 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" Nov 03 20:25:06.549202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.548827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj" event={"ID":"a23e8318-e623-4dd1-ba53-87eac194fbcf","Type":"ContainerDied","Data":"b7418a6ad6dc2aca9ee08ac2453cdc670f16f930f72299a725f1ea6655c3c9a6"} Nov 03 20:25:06.549202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.548871 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7418a6ad6dc2aca9ee08ac2453cdc670f16f930f72299a725f1ea6655c3c9a6" Nov 03 20:25:06.811467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.811323 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:06.947536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.947479 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca\") pod \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " Nov 03 20:25:06.947733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.947718 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca\") pod \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " Nov 03 20:25:06.947781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.947770 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj\" (UniqueName: \"kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj\") pod \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\" (UID: \"40fb2a69-834a-47fd-91dd-95f09b5b4c5f\") " Nov 03 20:25:06.947825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.947793 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca" pod "40fb2a69-834a-47fd-91dd-95f09b5b4c5f" (UID: "40fb2a69-834a-47fd-91dd-95f09b5b4c5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:06.948087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.948071 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:06.949902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.949882 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca" pod "40fb2a69-834a-47fd-91dd-95f09b5b4c5f" (UID: "40fb2a69-834a-47fd-91dd-95f09b5b4c5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:06.949981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:06.949900 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj" pod "40fb2a69-834a-47fd-91dd-95f09b5b4c5f" (UID: "40fb2a69-834a-47fd-91dd-95f09b5b4c5f"). InnerVolumeSpecName "kube-api-access-hnskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:07.049163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:07.049125 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:07.049301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:07.049185 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnskj\" (UniqueName: \"kubernetes.io/projected/40fb2a69-834a-47fd-91dd-95f09b5b4c5f-kube-api-access-hnskj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:07.557287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:07.557259 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" Nov 03 20:25:07.557735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:07.557405 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc" event={"ID":"40fb2a69-834a-47fd-91dd-95f09b5b4c5f","Type":"ContainerDied","Data":"9a4c8573e9bc9112e4e36bc5b0e0236da6541906e67bdbd777730addb107b677"} Nov 03 20:25:07.557735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:07.557466 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a4c8573e9bc9112e4e36bc5b0e0236da6541906e67bdbd777730addb107b677" Nov 03 20:25:10.524286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.524252 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x"] Nov 03 20:25:10.525091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.525073 5815 cpu_manager.go:414] "containerMap: removing container" podUID="40fb2a69-834a-47fd-91dd-95f09b5b4c5f" containerName="revoke-keys" Nov 03 20:25:10.525142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.525095 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fb2a69-834a-47fd-91dd-95f09b5b4c5f" containerName="revoke-keys" Nov 03 20:25:10.525176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.525141 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a23e8318-e623-4dd1-ba53-87eac194fbcf" containerName="revoke-keys" Nov 03 20:25:10.525176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.525149 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23e8318-e623-4dd1-ba53-87eac194fbcf" containerName="revoke-keys" Nov 03 20:25:10.525328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.525315 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fb2a69-834a-47fd-91dd-95f09b5b4c5f" containerName="revoke-keys" Nov 03 20:25:10.525363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.525340 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23e8318-e623-4dd1-ba53-87eac194fbcf" containerName="revoke-keys" Nov 03 20:25:10.529498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.529479 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.542349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.542323 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x"] Nov 03 20:25:10.629281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629239 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629412 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629551 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splzg\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629582 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629600 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629627 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629756 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629795 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.629846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.629820 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736407 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736536 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splzg\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736573 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736598 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736636 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736728 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.736802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736783 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.737084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736821 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.737084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.736861 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.770551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.757906 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.770551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.758174 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.770551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.758797 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.770551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.759033 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.770551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.759242 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.835630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.823714 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2"] Nov 03 20:25:10.905828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.898624 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:10.909985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.909956 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:10.910957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.910699 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:25:10.914943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.914912 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2"] Nov 03 20:25:10.917347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.917314 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.917483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.917465 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.920480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.920461 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:10.921417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:10.921376 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-splzg\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:11.054568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.054402 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdrn\" (UniqueName: \"kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.054781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.054590 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.054781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.054718 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.161577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.152725 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:11.161577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.157540 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdrn\" (UniqueName: \"kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.161577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.157745 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.161577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.157925 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.161577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.159103 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.177011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.176937 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.195232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.195140 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdrn\" (UniqueName: \"kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn\") pod \"maas-api-revoke-sub-3d7ecd61-xqmd2\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.254543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.253564 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:11.683581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.674934 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x"] Nov 03 20:25:11.746968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.746928 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:25:11.750155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.749231 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:25:11.750155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.749284 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:25:11.893534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:11.874139 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2"] Nov 03 20:25:12.794534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:12.793241 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" event={"ID":"3330058a-cc34-4507-b075-69b042759aff","Type":"ContainerStarted","Data":"4965d3324937e84f7f3936cc6f89f2a3ac9e22e3568225ba0f23436ec1e13a1c"} Nov 03 20:25:12.794534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:12.793310 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" event={"ID":"3330058a-cc34-4507-b075-69b042759aff","Type":"ContainerStarted","Data":"433ac4ca2202c281efcaa91bae3ae9946c0dec244949e2f4351d8b4632d874ea"} Nov 03 20:25:12.859529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:12.845640 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" event={"ID":"9dbf43e9-9dec-44d3-80b3-ef123df87ebb","Type":"ContainerStarted","Data":"cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608"} Nov 03 20:25:12.859529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:12.845709 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" event={"ID":"9dbf43e9-9dec-44d3-80b3-ef123df87ebb","Type":"ContainerStarted","Data":"39c91d95a07facda4aaac7e599e7d68854b4ec77a43186867ae36132607e9be2"} Nov 03 20:25:12.867543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:12.866312 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" podStartSLOduration=2.866291276 podStartE2EDuration="2.866291276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:12.83191631 +0000 UTC m=+1422.362261984" watchObservedRunningTime="2026-11-03 20:25:12.866291276 +0000 UTC m=+1422.396636949" Nov 03 20:25:12.918109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:12.918030 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podStartSLOduration=2.9180096559999997 podStartE2EDuration="2.918009656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:12.876690648 +0000 UTC m=+1422.407036323" watchObservedRunningTime="2026-11-03 20:25:12.918009656 +0000 UTC m=+1422.448355330" Nov 03 20:25:13.164932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:13.164902 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:13.186532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:13.165302 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:13.186819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:13.186786 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:13.908573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:13.907558 5815 generic.go:356] "Generic (PLEG): container finished" podID="3330058a-cc34-4507-b075-69b042759aff" containerID="4965d3324937e84f7f3936cc6f89f2a3ac9e22e3568225ba0f23436ec1e13a1c" exitCode=0 Nov 03 20:25:13.909064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:13.908922 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" event={"ID":"3330058a-cc34-4507-b075-69b042759aff","Type":"ContainerDied","Data":"4965d3324937e84f7f3936cc6f89f2a3ac9e22e3568225ba0f23436ec1e13a1c"} Nov 03 20:25:14.161870 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:25:14.161780 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:25:14.162556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:14.162529 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:14.162725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:14.162704 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:15.160158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.160123 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:15.160638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.160189 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:15.337578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.337154 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:15.457951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.457919 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca\") pod \"3330058a-cc34-4507-b075-69b042759aff\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " Nov 03 20:25:15.458116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.457972 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca\") pod \"3330058a-cc34-4507-b075-69b042759aff\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " Nov 03 20:25:15.458161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.458122 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn\" (UniqueName: \"kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn\") pod \"3330058a-cc34-4507-b075-69b042759aff\" (UID: \"3330058a-cc34-4507-b075-69b042759aff\") " Nov 03 20:25:15.458321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.458295 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca" pod "3330058a-cc34-4507-b075-69b042759aff" (UID: "3330058a-cc34-4507-b075-69b042759aff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:15.458559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.458539 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3330058a-cc34-4507-b075-69b042759aff-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:15.460215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.460198 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca" pod "3330058a-cc34-4507-b075-69b042759aff" (UID: "3330058a-cc34-4507-b075-69b042759aff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:15.460296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.460277 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn" pod "3330058a-cc34-4507-b075-69b042759aff" (UID: "3330058a-cc34-4507-b075-69b042759aff"). InnerVolumeSpecName "kube-api-access-bjdrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:15.559477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.559440 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdrn\" (UniqueName: \"kubernetes.io/projected/3330058a-cc34-4507-b075-69b042759aff-kube-api-access-bjdrn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:15.559477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.559468 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3330058a-cc34-4507-b075-69b042759aff-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:16.002558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.999372 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" event={"ID":"3330058a-cc34-4507-b075-69b042759aff","Type":"ContainerDied","Data":"433ac4ca2202c281efcaa91bae3ae9946c0dec244949e2f4351d8b4632d874ea"} Nov 03 20:25:16.002558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.999432 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433ac4ca2202c281efcaa91bae3ae9946c0dec244949e2f4351d8b4632d874ea" Nov 03 20:25:16.002558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:15.999525 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2" Nov 03 20:25:16.159125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:16.159094 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:16.159330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:16.159151 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:17.159415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:17.159374 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:17.159859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:17.159437 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:18.160135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:18.160098 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:18.160575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:18.160156 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:19.168931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:19.159873 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:19.168931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:19.159949 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:25:20.391722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:20.391687 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:20.392825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:20.392788 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:20.393638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:20.393613 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:25:21.897536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.892898 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v"] Nov 03 20:25:21.897536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.893890 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3330058a-cc34-4507-b075-69b042759aff" containerName="revoke-keys" Nov 03 20:25:21.897536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.893912 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3330058a-cc34-4507-b075-69b042759aff" containerName="revoke-keys" Nov 03 20:25:21.897536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.894126 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3330058a-cc34-4507-b075-69b042759aff" containerName="revoke-keys" Nov 03 20:25:21.926278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.926246 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v"] Nov 03 20:25:21.926446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.926376 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:21.930976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.930944 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:21.932067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:21.932047 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:25:22.004857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.004823 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96h84\" (UniqueName: \"kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.005036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.004953 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.005102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.005075 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.105872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.105844 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96h84\" (UniqueName: \"kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.106042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.105919 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.106042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.106012 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.106677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.106656 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.108582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.108564 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.118353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.118330 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96h84\" (UniqueName: \"kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84\") pod \"maas-api-revoke-sub-b0d53621-9272v\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.243401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.243377 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:22.484585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.484550 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2"] Nov 03 20:25:22.488891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.488874 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.497776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.497715 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2"] Nov 03 20:25:22.538656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.538627 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v"] Nov 03 20:25:22.539809 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:22.539782 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecb1f63_8989_448f_a648_32d0e677d708.slice/crio-0fae46067d9ca237c4df31ef81d9443a617f27f05e361d49a10657c3e5749151 WatchSource:0}: Error finding container 0fae46067d9ca237c4df31ef81d9443a617f27f05e361d49a10657c3e5749151: Status 404 returned error can't find the container with id 0fae46067d9ca237c4df31ef81d9443a617f27f05e361d49a10657c3e5749151 Nov 03 20:25:22.595535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.595491 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2"] Nov 03 20:25:22.601954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.601930 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.607816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.607792 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2"] Nov 03 20:25:22.676682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.676651 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.683446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.683390 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxgnm\" (UniqueName: \"kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.683961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.683907 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.785965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.785931 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.786123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.786060 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtj8f\" (UniqueName: \"kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.786191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.786131 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.786449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.786421 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.786636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.786482 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxgnm\" (UniqueName: \"kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.786713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.786687 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.786973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.786951 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.789161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.789139 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.799107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.799089 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxgnm\" (UniqueName: \"kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm\") pod \"maas-api-revoke-sub-ddb34247-wj4g2\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.814630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.814610 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:22.892142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.888496 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.892142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.889185 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtj8f\" (UniqueName: \"kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.892142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.889303 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.892142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.890275 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.895581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.895536 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.903191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.903144 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtj8f\" (UniqueName: \"kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f\") pod \"maas-api-revoke-sub-82505810-jrht2\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:22.996320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:22.996279 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:23.070578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:23.066695 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" event={"ID":"cecb1f63-8989-448f-a648-32d0e677d708","Type":"ContainerStarted","Data":"2ee94bc5d3c07937c5029680301adb40db33aae3b391e9cf3f57e4801ff8ab97"} Nov 03 20:25:23.070578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:23.066759 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" event={"ID":"cecb1f63-8989-448f-a648-32d0e677d708","Type":"ContainerStarted","Data":"0fae46067d9ca237c4df31ef81d9443a617f27f05e361d49a10657c3e5749151"} Nov 03 20:25:23.087082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:23.087029 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" podStartSLOduration=2.087007519 podStartE2EDuration="2.087007519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:23.085782493 +0000 UTC m=+1432.616128165" watchObservedRunningTime="2026-11-03 20:25:23.087007519 +0000 UTC m=+1432.617353219" Nov 03 20:25:23.109827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:23.109782 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2"] Nov 03 20:25:23.277832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:23.277805 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2"] Nov 03 20:25:23.278795 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:23.278765 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3114265f_061d_4a4d_b6c6_e7b1b1e215d1.slice/crio-71a43d9e9ff895728af5edc75a4b420f5b633b94e7fdf7088a7b4b4a1b1f3631 WatchSource:0}: Error finding container 71a43d9e9ff895728af5edc75a4b420f5b633b94e7fdf7088a7b4b4a1b1f3631: Status 404 returned error can't find the container with id 71a43d9e9ff895728af5edc75a4b420f5b633b94e7fdf7088a7b4b4a1b1f3631 Nov 03 20:25:24.079566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.079537 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" event={"ID":"dc1cff36-3f99-4fbf-83e0-3802126ab5df","Type":"ContainerStarted","Data":"0d621fa9a9899f1ce0785c9005e642d6ec7b39f015b9038bb55667aa4ca81fe3"} Nov 03 20:25:24.079975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.079600 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" event={"ID":"dc1cff36-3f99-4fbf-83e0-3802126ab5df","Type":"ContainerStarted","Data":"c1ceccdd8654bac9f24caddb060ce13fc2802d5e9d0eea0ff5df83fecf81fd77"} Nov 03 20:25:24.082167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.082136 5815 generic.go:356] "Generic (PLEG): container finished" podID="cecb1f63-8989-448f-a648-32d0e677d708" containerID="2ee94bc5d3c07937c5029680301adb40db33aae3b391e9cf3f57e4801ff8ab97" exitCode=0 Nov 03 20:25:24.082324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.082302 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" event={"ID":"cecb1f63-8989-448f-a648-32d0e677d708","Type":"ContainerDied","Data":"2ee94bc5d3c07937c5029680301adb40db33aae3b391e9cf3f57e4801ff8ab97"} Nov 03 20:25:24.084769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.084740 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" event={"ID":"3114265f-061d-4a4d-b6c6-e7b1b1e215d1","Type":"ContainerStarted","Data":"7b528e1b42fd15aaea4a783b22dea3fcc44aeae0ead8fe76c5bcf1e5abb9bd83"} Nov 03 20:25:24.084881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.084798 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" event={"ID":"3114265f-061d-4a4d-b6c6-e7b1b1e215d1","Type":"ContainerStarted","Data":"71a43d9e9ff895728af5edc75a4b420f5b633b94e7fdf7088a7b4b4a1b1f3631"} Nov 03 20:25:24.099956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.099914 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" podStartSLOduration=2.099902503 podStartE2EDuration="2.099902503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:24.097537878 +0000 UTC m=+1433.627883550" watchObservedRunningTime="2026-11-03 20:25:24.099902503 +0000 UTC m=+1433.630248174" Nov 03 20:25:24.143577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:24.143522 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" podStartSLOduration=2.143487292 podStartE2EDuration="2.143487292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:24.139646789 +0000 UTC m=+1433.669992460" watchObservedRunningTime="2026-11-03 20:25:24.143487292 +0000 UTC m=+1433.673832965" Nov 03 20:25:25.096371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.096331 5815 generic.go:356] "Generic (PLEG): container finished" podID="dc1cff36-3f99-4fbf-83e0-3802126ab5df" containerID="0d621fa9a9899f1ce0785c9005e642d6ec7b39f015b9038bb55667aa4ca81fe3" exitCode=0 Nov 03 20:25:25.096948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.096429 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" event={"ID":"dc1cff36-3f99-4fbf-83e0-3802126ab5df","Type":"ContainerDied","Data":"0d621fa9a9899f1ce0785c9005e642d6ec7b39f015b9038bb55667aa4ca81fe3"} Nov 03 20:25:25.098759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.098734 5815 generic.go:356] "Generic (PLEG): container finished" podID="3114265f-061d-4a4d-b6c6-e7b1b1e215d1" containerID="7b528e1b42fd15aaea4a783b22dea3fcc44aeae0ead8fe76c5bcf1e5abb9bd83" exitCode=0 Nov 03 20:25:25.098891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.098806 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" event={"ID":"3114265f-061d-4a4d-b6c6-e7b1b1e215d1","Type":"ContainerDied","Data":"7b528e1b42fd15aaea4a783b22dea3fcc44aeae0ead8fe76c5bcf1e5abb9bd83"} Nov 03 20:25:25.275040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.275014 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:25.330431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.330396 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca\") pod \"cecb1f63-8989-448f-a648-32d0e677d708\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " Nov 03 20:25:25.330678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.330657 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84\" (UniqueName: \"kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84\") pod \"cecb1f63-8989-448f-a648-32d0e677d708\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " Nov 03 20:25:25.330844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.330826 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca\") pod \"cecb1f63-8989-448f-a648-32d0e677d708\" (UID: \"cecb1f63-8989-448f-a648-32d0e677d708\") " Nov 03 20:25:25.330922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.330861 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca" pod "cecb1f63-8989-448f-a648-32d0e677d708" (UID: "cecb1f63-8989-448f-a648-32d0e677d708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:25.331479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.331424 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:25.333056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.333030 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca" pod "cecb1f63-8989-448f-a648-32d0e677d708" (UID: "cecb1f63-8989-448f-a648-32d0e677d708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:25.333493 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.333472 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84" pod "cecb1f63-8989-448f-a648-32d0e677d708" (UID: "cecb1f63-8989-448f-a648-32d0e677d708"). InnerVolumeSpecName "kube-api-access-96h84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:25.432965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.432865 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96h84\" (UniqueName: \"kubernetes.io/projected/cecb1f63-8989-448f-a648-32d0e677d708-kube-api-access-96h84\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:25.432965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:25.432895 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cecb1f63-8989-448f-a648-32d0e677d708-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.109858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.109826 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" Nov 03 20:25:26.110341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.109858 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v" event={"ID":"cecb1f63-8989-448f-a648-32d0e677d708","Type":"ContainerDied","Data":"0fae46067d9ca237c4df31ef81d9443a617f27f05e361d49a10657c3e5749151"} Nov 03 20:25:26.110341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.109928 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fae46067d9ca237c4df31ef81d9443a617f27f05e361d49a10657c3e5749151" Nov 03 20:25:26.330478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.330455 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:26.333025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.333005 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.456019 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca\") pod \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.456091 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca\") pod \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.456122 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca\") pod \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.456235 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f\" (UniqueName: \"kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f\") pod \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\" (UID: \"3114265f-061d-4a4d-b6c6-e7b1b1e215d1\") " Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.456293 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca\") pod \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.456398 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm\" (UniqueName: \"kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm\") pod \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\" (UID: \"dc1cff36-3f99-4fbf-83e0-3802126ab5df\") " Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.458240 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca" pod "3114265f-061d-4a4d-b6c6-e7b1b1e215d1" (UID: "3114265f-061d-4a4d-b6c6-e7b1b1e215d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:26.460586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.458647 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca" pod "dc1cff36-3f99-4fbf-83e0-3802126ab5df" (UID: "dc1cff36-3f99-4fbf-83e0-3802126ab5df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:26.489790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.489008 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f" pod "3114265f-061d-4a4d-b6c6-e7b1b1e215d1" (UID: "3114265f-061d-4a4d-b6c6-e7b1b1e215d1"). InnerVolumeSpecName "kube-api-access-vtj8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:26.491618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.490602 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca" pod "3114265f-061d-4a4d-b6c6-e7b1b1e215d1" (UID: "3114265f-061d-4a4d-b6c6-e7b1b1e215d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:26.494969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.494942 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca" pod "dc1cff36-3f99-4fbf-83e0-3802126ab5df" (UID: "dc1cff36-3f99-4fbf-83e0-3802126ab5df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:26.495242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.495218 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm" pod "dc1cff36-3f99-4fbf-83e0-3802126ab5df" (UID: "dc1cff36-3f99-4fbf-83e0-3802126ab5df"). InnerVolumeSpecName "kube-api-access-qxgnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:26.559423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.559393 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.559423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.559421 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.559666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.559431 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtj8f\" (UniqueName: \"kubernetes.io/projected/3114265f-061d-4a4d-b6c6-e7b1b1e215d1-kube-api-access-vtj8f\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.559666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.559443 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.559666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.559453 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxgnm\" (UniqueName: \"kubernetes.io/projected/dc1cff36-3f99-4fbf-83e0-3802126ab5df-kube-api-access-qxgnm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:26.559666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:26.559464 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cff36-3f99-4fbf-83e0-3802126ab5df-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:27.152183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:27.148257 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" Nov 03 20:25:27.158020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:27.156327 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" Nov 03 20:25:27.158020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:27.157186 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2" event={"ID":"dc1cff36-3f99-4fbf-83e0-3802126ab5df","Type":"ContainerDied","Data":"c1ceccdd8654bac9f24caddb060ce13fc2802d5e9d0eea0ff5df83fecf81fd77"} Nov 03 20:25:27.158020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:27.157236 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ceccdd8654bac9f24caddb060ce13fc2802d5e9d0eea0ff5df83fecf81fd77" Nov 03 20:25:27.158020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:27.157254 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2" event={"ID":"3114265f-061d-4a4d-b6c6-e7b1b1e215d1","Type":"ContainerDied","Data":"71a43d9e9ff895728af5edc75a4b420f5b633b94e7fdf7088a7b4b4a1b1f3631"} Nov 03 20:25:27.158020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:27.157276 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a43d9e9ff895728af5edc75a4b420f5b633b94e7fdf7088a7b4b4a1b1f3631" Nov 03 20:25:30.847490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:30.847458 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k"] Nov 03 20:25:32.144737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:32.144703 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:25:34.456535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:34.443644 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:34.569179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:34.569115 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:34.707130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:34.706481 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:34.737643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:34.737596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": context deadline exceeded" Nov 03 20:25:34.828533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:34.819609 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.003116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.003084 5815 scope.go:122] "RemoveContainer" containerID="5290f688bca335726cfd53640a450a361111dc66d862c1d4aa1f0fc710e1847b" Nov 03 20:25:35.093683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.093653 5815 scope.go:122] "RemoveContainer" containerID="ef2f1aa0656144c6009f1ce01fb3126b1effe1d5f8986ff2809787f59a12de79" Nov 03 20:25:35.137533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.133652 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.160649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.160602 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.163762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.163362 5815 scope.go:122] "RemoveContainer" containerID="20e3cb6e02cb37c33496154cf6a7df205e59773008a76a71b0a9bce95f056c2d" Nov 03 20:25:35.455785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.453219 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.455785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.454228 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:25:35.466547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.465662 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:25:35.591540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.576603 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.591540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.577368 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:25:35.708801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.708686 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:25:35.728539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.715882 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:25:35.765645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.745398 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.765645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.746139 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:25:35.831385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.831342 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:25:35.854086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:35.854054 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:25:36.206533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.199844 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:25:36.248397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.213972 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:25:36.476598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.460630 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:25:36.561187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.561135 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.644766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.580717 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:25:36.644766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.588313 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:25:36.685539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.665826 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:36.685539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.665882 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.786937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.732530 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:25:36.786937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.745596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.869723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:36.869655 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.097190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.097103 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf"] Nov 03 20:25:37.122540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.110201 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3114265f-061d-4a4d-b6c6-e7b1b1e215d1" containerName="revoke-keys" Nov 03 20:25:37.123216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.123189 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3114265f-061d-4a4d-b6c6-e7b1b1e215d1" containerName="revoke-keys" Nov 03 20:25:37.123400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.123387 5815 cpu_manager.go:414] "containerMap: removing container" podUID="cecb1f63-8989-448f-a648-32d0e677d708" containerName="revoke-keys" Nov 03 20:25:37.127530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.123489 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecb1f63-8989-448f-a648-32d0e677d708" containerName="revoke-keys" Nov 03 20:25:37.127674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.127659 5815 cpu_manager.go:414] "containerMap: removing container" podUID="dc1cff36-3f99-4fbf-83e0-3802126ab5df" containerName="revoke-keys" Nov 03 20:25:37.127763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.127750 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1cff36-3f99-4fbf-83e0-3802126ab5df" containerName="revoke-keys" Nov 03 20:25:37.147968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.147939 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3114265f-061d-4a4d-b6c6-e7b1b1e215d1" containerName="revoke-keys" Nov 03 20:25:37.148167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.148150 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecb1f63-8989-448f-a648-32d0e677d708" containerName="revoke-keys" Nov 03 20:25:37.148300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.148288 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1cff36-3f99-4fbf-83e0-3802126ab5df" containerName="revoke-keys" Nov 03 20:25:37.212531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.202788 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:25:37.212531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.202828 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf"] Nov 03 20:25:37.212531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.202895 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.212531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.205850 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.239533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.224536 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.239533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.224634 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.239533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.224691 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtdh\" (UniqueName: \"kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.375589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.348327 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:37.375589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.348451 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:25:37.397452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.395569 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.397452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.395734 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.397452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.395812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtdh\" (UniqueName: \"kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.397452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.397010 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.698783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.694683 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 20:25:37.698783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.696682 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 03 20:25:37.762862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.762814 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.782632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.782586 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.887565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.844310 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:25:37.887565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.844719 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:25:37.909455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.899554 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:37.909455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:37.903375 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.093543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.085032 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtdh\" (UniqueName: \"kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh\") pod \"maas-api-revoke-sub-a38de8d6-jgdgf\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:38.209581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.206680 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.264275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.264238 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:38.264586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.264552 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.354535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.353943 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:38.443833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.443800 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:38.444065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.444038 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.582563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.573107 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:25:38.805601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.787141 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" probeResult="failure" output="Get \"https://10.132.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.871567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.862977 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.871567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.863333 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.875457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.875429 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:38.875708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.875681 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.916529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:38.911073 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:39.070261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.068676 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:39.172732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.172554 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:25:39.267296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.267254 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:39.272301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.269756 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:25:39.352590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.352454 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:25:39.359670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.359642 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:25:39.393158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.393106 5815 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:39.393410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.393184 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:39.468445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.468365 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:25:39.797035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.797002 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf"] Nov 03 20:25:39.817473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.815997 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2"] Nov 03 20:25:39.824564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.824541 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:39.830456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.830351 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2"] Nov 03 20:25:39.945081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.944999 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:39.945257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.945152 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:39.945364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:39.945345 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94v2n\" (UniqueName: \"kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.057533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.055546 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.057533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.055722 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94v2n\" (UniqueName: \"kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.057533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.055838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.057533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.056725 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.093544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.092426 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.113175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.113079 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" event={"ID":"9d546894-0603-4437-acfe-56632291e361","Type":"ContainerStarted","Data":"5737b645b7b3bce299d76d18545a097e54081f84452bd9135b0a51fd67dad2d8"} Nov 03 20:25:40.113750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.113729 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94v2n\" (UniqueName: \"kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n\") pod \"maas-api-revoke-sub-b5fa8100-225t2\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.169879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.169856 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:40.310487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.310452 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh"] Nov 03 20:25:40.314803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.314778 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.322752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.322724 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh"] Nov 03 20:25:40.457910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.457879 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2"] Nov 03 20:25:40.458231 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:40.458207 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7edac27_bd6a_4c3d_8e40_9d7230628363.slice/crio-27b89856b906d1665380670afa006db316d5dbf2be4eb83308047f7349f5e9a3 WatchSource:0}: Error finding container 27b89856b906d1665380670afa006db316d5dbf2be4eb83308047f7349f5e9a3: Status 404 returned error can't find the container with id 27b89856b906d1665380670afa006db316d5dbf2be4eb83308047f7349f5e9a3 Nov 03 20:25:40.460605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.460580 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwg5\" (UniqueName: \"kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.460814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.460789 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.460878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.460831 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.562250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.562222 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwg5\" (UniqueName: \"kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.562359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.562288 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.562359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.562305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.563016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.562994 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.564931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.564911 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.574211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.574183 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwg5\" (UniqueName: \"kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5\") pod \"maas-api-revoke-sub-b707df2c-r2jjh\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.632997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.632959 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:40.918310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:40.918288 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh"] Nov 03 20:25:40.925668 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:40.925642 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec51a1f_f96e_4436_8e9d_b3471beabd32.slice/crio-8806ef00c9e9626094bc5f359c09b65a4868ea5cb64e82bc59abf39740d70a60 WatchSource:0}: Error finding container 8806ef00c9e9626094bc5f359c09b65a4868ea5cb64e82bc59abf39740d70a60: Status 404 returned error can't find the container with id 8806ef00c9e9626094bc5f359c09b65a4868ea5cb64e82bc59abf39740d70a60 Nov 03 20:25:41.122785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:41.122747 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" event={"ID":"a7edac27-bd6a-4c3d-8e40-9d7230628363","Type":"ContainerStarted","Data":"4d1d691304eacd6afb959c526764a487f57224ac2686e53a1af20b41c3c3fb8a"} Nov 03 20:25:41.122984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:41.122970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" event={"ID":"a7edac27-bd6a-4c3d-8e40-9d7230628363","Type":"ContainerStarted","Data":"27b89856b906d1665380670afa006db316d5dbf2be4eb83308047f7349f5e9a3"} Nov 03 20:25:41.124324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:41.124301 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" event={"ID":"bec51a1f-f96e-4436-8e9d-b3471beabd32","Type":"ContainerStarted","Data":"8806ef00c9e9626094bc5f359c09b65a4868ea5cb64e82bc59abf39740d70a60"} Nov 03 20:25:41.125998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:41.125974 5815 generic.go:356] "Generic (PLEG): container finished" podID="9d546894-0603-4437-acfe-56632291e361" containerID="98a381b0aa8e1e5793f43710a2bd533ef4365f303f0140548929a1155b6507f8" exitCode=0 Nov 03 20:25:41.126117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:41.126077 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" event={"ID":"9d546894-0603-4437-acfe-56632291e361","Type":"ContainerDied","Data":"98a381b0aa8e1e5793f43710a2bd533ef4365f303f0140548929a1155b6507f8"} Nov 03 20:25:41.144069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:41.144031 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" podStartSLOduration=2.14402073 podStartE2EDuration="2.14402073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:41.140475084 +0000 UTC m=+1450.670820756" watchObservedRunningTime="2026-11-03 20:25:41.14402073 +0000 UTC m=+1450.674366402" Nov 03 20:25:42.135636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.135558 5815 generic.go:356] "Generic (PLEG): container finished" podID="a7edac27-bd6a-4c3d-8e40-9d7230628363" containerID="4d1d691304eacd6afb959c526764a487f57224ac2686e53a1af20b41c3c3fb8a" exitCode=0 Nov 03 20:25:42.136098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.135645 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" event={"ID":"a7edac27-bd6a-4c3d-8e40-9d7230628363","Type":"ContainerDied","Data":"4d1d691304eacd6afb959c526764a487f57224ac2686e53a1af20b41c3c3fb8a"} Nov 03 20:25:42.137701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.137669 5815 generic.go:356] "Generic (PLEG): container finished" podID="bec51a1f-f96e-4436-8e9d-b3471beabd32" containerID="b70930781e02f5dd9a683b4188f0c42cee154840834199507ffc09b97127a7a1" exitCode=0 Nov 03 20:25:42.137818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.137750 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" event={"ID":"bec51a1f-f96e-4436-8e9d-b3471beabd32","Type":"ContainerDied","Data":"b70930781e02f5dd9a683b4188f0c42cee154840834199507ffc09b97127a7a1"} Nov 03 20:25:42.328440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.328413 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:42.511191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.511154 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca\") pod \"9d546894-0603-4437-acfe-56632291e361\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " Nov 03 20:25:42.511404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.511244 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh\" (UniqueName: \"kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh\") pod \"9d546894-0603-4437-acfe-56632291e361\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " Nov 03 20:25:42.511404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.511372 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca\") pod \"9d546894-0603-4437-acfe-56632291e361\" (UID: \"9d546894-0603-4437-acfe-56632291e361\") " Nov 03 20:25:42.511952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.511916 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca" pod "9d546894-0603-4437-acfe-56632291e361" (UID: "9d546894-0603-4437-acfe-56632291e361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:42.513868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.513842 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca" pod "9d546894-0603-4437-acfe-56632291e361" (UID: "9d546894-0603-4437-acfe-56632291e361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:42.513991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.513911 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh" pod "9d546894-0603-4437-acfe-56632291e361" (UID: "9d546894-0603-4437-acfe-56632291e361"). InnerVolumeSpecName "kube-api-access-7mtdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:42.613182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.613150 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d546894-0603-4437-acfe-56632291e361-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:42.613182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.613178 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtdh\" (UniqueName: \"kubernetes.io/projected/9d546894-0603-4437-acfe-56632291e361-kube-api-access-7mtdh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:42.613350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:42.613189 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d546894-0603-4437-acfe-56632291e361-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.147359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.147332 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" Nov 03 20:25:43.148277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.147367 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf" event={"ID":"9d546894-0603-4437-acfe-56632291e361","Type":"ContainerDied","Data":"5737b645b7b3bce299d76d18545a097e54081f84452bd9135b0a51fd67dad2d8"} Nov 03 20:25:43.148277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.147419 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5737b645b7b3bce299d76d18545a097e54081f84452bd9135b0a51fd67dad2d8" Nov 03 20:25:43.322339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.322316 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:43.352088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.352068 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:43.425471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.425395 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca\") pod \"bec51a1f-f96e-4436-8e9d-b3471beabd32\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " Nov 03 20:25:43.425471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.425451 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5\" (UniqueName: \"kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5\") pod \"bec51a1f-f96e-4436-8e9d-b3471beabd32\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " Nov 03 20:25:43.425471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.425469 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca\") pod \"bec51a1f-f96e-4436-8e9d-b3471beabd32\" (UID: \"bec51a1f-f96e-4436-8e9d-b3471beabd32\") " Nov 03 20:25:43.425837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.425811 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca" pod "bec51a1f-f96e-4436-8e9d-b3471beabd32" (UID: "bec51a1f-f96e-4436-8e9d-b3471beabd32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:43.426089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.426072 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.427745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.427726 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca" pod "bec51a1f-f96e-4436-8e9d-b3471beabd32" (UID: "bec51a1f-f96e-4436-8e9d-b3471beabd32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:43.427884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.427867 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5" pod "bec51a1f-f96e-4436-8e9d-b3471beabd32" (UID: "bec51a1f-f96e-4436-8e9d-b3471beabd32"). InnerVolumeSpecName "kube-api-access-crwg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:43.527220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527194 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca\") pod \"a7edac27-bd6a-4c3d-8e40-9d7230628363\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " Nov 03 20:25:43.527399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527387 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca\") pod \"a7edac27-bd6a-4c3d-8e40-9d7230628363\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " Nov 03 20:25:43.527461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527444 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n\" (UniqueName: \"kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n\") pod \"a7edac27-bd6a-4c3d-8e40-9d7230628363\" (UID: \"a7edac27-bd6a-4c3d-8e40-9d7230628363\") " Nov 03 20:25:43.527537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527517 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca" pod "a7edac27-bd6a-4c3d-8e40-9d7230628363" (UID: "a7edac27-bd6a-4c3d-8e40-9d7230628363"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:43.527813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527798 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.527877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527816 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwg5\" (UniqueName: \"kubernetes.io/projected/bec51a1f-f96e-4436-8e9d-b3471beabd32-kube-api-access-crwg5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.527877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.527826 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bec51a1f-f96e-4436-8e9d-b3471beabd32-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.529459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.529437 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca" pod "a7edac27-bd6a-4c3d-8e40-9d7230628363" (UID: "a7edac27-bd6a-4c3d-8e40-9d7230628363"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:43.529831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.529811 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n" pod "a7edac27-bd6a-4c3d-8e40-9d7230628363" (UID: "a7edac27-bd6a-4c3d-8e40-9d7230628363"). InnerVolumeSpecName "kube-api-access-94v2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:43.628895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.628863 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7edac27-bd6a-4c3d-8e40-9d7230628363-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.629043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.628898 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94v2n\" (UniqueName: \"kubernetes.io/projected/a7edac27-bd6a-4c3d-8e40-9d7230628363-kube-api-access-94v2n\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:43.863971 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:25:43.863926 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:25:43.880539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.880476 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd"] Nov 03 20:25:43.881163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881142 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a7edac27-bd6a-4c3d-8e40-9d7230628363" containerName="revoke-keys" Nov 03 20:25:43.881163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881162 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7edac27-bd6a-4c3d-8e40-9d7230628363" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881178 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bec51a1f-f96e-4436-8e9d-b3471beabd32" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881183 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec51a1f-f96e-4436-8e9d-b3471beabd32" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881191 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9d546894-0603-4437-acfe-56632291e361" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881196 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d546894-0603-4437-acfe-56632291e361" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881326 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7edac27-bd6a-4c3d-8e40-9d7230628363" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881335 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec51a1f-f96e-4436-8e9d-b3471beabd32" containerName="revoke-keys" Nov 03 20:25:43.881377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.881341 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d546894-0603-4437-acfe-56632291e361" containerName="revoke-keys" Nov 03 20:25:43.885695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.885676 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:43.899227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:43.899018 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd"] Nov 03 20:25:44.036468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.036436 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfbd\" (UniqueName: \"kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.036679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.036661 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.036861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.036838 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.137903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.137826 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.138050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.137956 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.138050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.138010 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfbd\" (UniqueName: \"kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.138765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.138734 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.140694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.140674 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.152906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.152887 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfbd\" (UniqueName: \"kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd\") pod \"maas-api-revoke-sub-a4be838b-jzcnd\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.157190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.157174 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" Nov 03 20:25:44.157284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.157204 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh" event={"ID":"bec51a1f-f96e-4436-8e9d-b3471beabd32","Type":"ContainerDied","Data":"8806ef00c9e9626094bc5f359c09b65a4868ea5cb64e82bc59abf39740d70a60"} Nov 03 20:25:44.157284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.157247 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8806ef00c9e9626094bc5f359c09b65a4868ea5cb64e82bc59abf39740d70a60" Nov 03 20:25:44.158710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.158688 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" event={"ID":"a7edac27-bd6a-4c3d-8e40-9d7230628363","Type":"ContainerDied","Data":"27b89856b906d1665380670afa006db316d5dbf2be4eb83308047f7349f5e9a3"} Nov 03 20:25:44.158710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.158703 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2" Nov 03 20:25:44.158847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.158727 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b89856b906d1665380670afa006db316d5dbf2be4eb83308047f7349f5e9a3" Nov 03 20:25:44.216266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.216243 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:44.500704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:44.500666 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd"] Nov 03 20:25:44.500840 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:25:44.500791 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2c7c60_8564_4bef_be38_3dd6c7ba03a4.slice/crio-c7e5c0e0d7cbd2cd54e9682bab88a3b3b64545f9edd2b5e3f0e77c8281663619 WatchSource:0}: Error finding container c7e5c0e0d7cbd2cd54e9682bab88a3b3b64545f9edd2b5e3f0e77c8281663619: Status 404 returned error can't find the container with id c7e5c0e0d7cbd2cd54e9682bab88a3b3b64545f9edd2b5e3f0e77c8281663619 Nov 03 20:25:45.169061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:45.168981 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" event={"ID":"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4","Type":"ContainerStarted","Data":"14574acc1fecf1f8e886b2417e0b68a05db8872dce1ea37b854822d9edd4e1db"} Nov 03 20:25:45.169061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:45.169032 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" event={"ID":"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4","Type":"ContainerStarted","Data":"c7e5c0e0d7cbd2cd54e9682bab88a3b3b64545f9edd2b5e3f0e77c8281663619"} Nov 03 20:25:45.193054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:45.192996 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" podStartSLOduration=2.192978115 podStartE2EDuration="2.192978115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:45.186721358 +0000 UTC m=+1454.717067030" watchObservedRunningTime="2026-11-03 20:25:45.192978115 +0000 UTC m=+1454.723323789" Nov 03 20:25:45.870567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:45.870419 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" containerID="cri-o://cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508" gracePeriod=45 Nov 03 20:25:46.146218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.146188 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:25:46.181321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.181290 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache\") pod \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " Nov 03 20:25:46.181852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.181390 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir\") pod \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " Nov 03 20:25:46.181852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.181452 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home\") pod \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " Nov 03 20:25:46.181852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.181555 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm\") pod \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " Nov 03 20:25:46.181852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.181682 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf\" (UniqueName: \"kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf\") pod \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " Nov 03 20:25:46.181852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.181847 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs\") pod \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\" (UID: \"1b63215f-9fd3-49d3-a6c5-c1104fe44376\") " Nov 03 20:25:46.185105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.185061 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache" pod "1b63215f-9fd3-49d3-a6c5-c1104fe44376" (UID: "1b63215f-9fd3-49d3-a6c5-c1104fe44376"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:46.185255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.185223 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir" pod "1b63215f-9fd3-49d3-a6c5-c1104fe44376" (UID: "1b63215f-9fd3-49d3-a6c5-c1104fe44376"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:46.185709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.185403 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home" pod "1b63215f-9fd3-49d3-a6c5-c1104fe44376" (UID: "1b63215f-9fd3-49d3-a6c5-c1104fe44376"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:46.185709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.185467 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs" pod "1b63215f-9fd3-49d3-a6c5-c1104fe44376" (UID: "1b63215f-9fd3-49d3-a6c5-c1104fe44376"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:46.187259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.187225 5815 generic.go:356] "Generic (PLEG): container finished" podID="3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" containerID="14574acc1fecf1f8e886b2417e0b68a05db8872dce1ea37b854822d9edd4e1db" exitCode=0 Nov 03 20:25:46.187366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.187329 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" event={"ID":"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4","Type":"ContainerDied","Data":"14574acc1fecf1f8e886b2417e0b68a05db8872dce1ea37b854822d9edd4e1db"} Nov 03 20:25:46.188276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.188251 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm" pod "1b63215f-9fd3-49d3-a6c5-c1104fe44376" (UID: "1b63215f-9fd3-49d3-a6c5-c1104fe44376"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:46.189020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.188988 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf" pod "1b63215f-9fd3-49d3-a6c5-c1104fe44376" (UID: "1b63215f-9fd3-49d3-a6c5-c1104fe44376"). InnerVolumeSpecName "kube-api-access-hmxtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:46.189743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.189717 5815 generic.go:356] "Generic (PLEG): container finished" podID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerID="cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508" exitCode=0 Nov 03 20:25:46.189848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.189811 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" event={"ID":"1b63215f-9fd3-49d3-a6c5-c1104fe44376","Type":"ContainerDied","Data":"cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508"} Nov 03 20:25:46.189848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.189839 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" event={"ID":"1b63215f-9fd3-49d3-a6c5-c1104fe44376","Type":"ContainerDied","Data":"d6c6bc3c2200b9995c803854462bbeda663ef0a3da07b66596d732303926333d"} Nov 03 20:25:46.189957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.189861 5815 scope.go:122] "RemoveContainer" containerID="cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508" Nov 03 20:25:46.190006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.189991 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k" Nov 03 20:25:46.222316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.222258 5815 scope.go:122] "RemoveContainer" containerID="cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508" Nov 03 20:25:46.230579 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:25:46.229404 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508\": container with ID starting with cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508 not found: ID does not exist" containerID="cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508" Nov 03 20:25:46.230579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.229459 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508"} err="failed to get container status \"cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508\": rpc error: code = NotFound desc = could not find container \"cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508\": container with ID starting with cccba4de7599dd3e93c65c318e0f6ec086a48ddc8e4af2d508d2444a55b90508 not found: ID does not exist" Nov 03 20:25:46.237639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.237609 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k"] Nov 03 20:25:46.241263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.241237 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2eee8c/ipp-route-2eee8c-kserve-55d4959f6f-65w8k"] Nov 03 20:25:46.288534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.286733 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.288534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.286770 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.288534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.286787 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxtf\" (UniqueName: \"kubernetes.io/projected/1b63215f-9fd3-49d3-a6c5-c1104fe44376-kube-api-access-hmxtf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.288534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.286802 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.288534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.286818 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.288534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.286831 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b63215f-9fd3-49d3-a6c5-c1104fe44376-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:46.587796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:46.587766 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" path="/var/lib/kubelet/pods/1b63215f-9fd3-49d3-a6c5-c1104fe44376/volumes" Nov 03 20:25:47.394523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.394485 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:47.506777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.506741 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca\") pod \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " Nov 03 20:25:47.506954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.506836 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca\") pod \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " Nov 03 20:25:47.506954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.506866 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd\" (UniqueName: \"kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd\") pod \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\" (UID: \"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4\") " Nov 03 20:25:47.507170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.507148 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca" pod "3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" (UID: "3d2c7c60-8564-4bef-be38-3dd6c7ba03a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:47.507416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.507392 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:47.509009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.508980 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca" pod "3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" (UID: "3d2c7c60-8564-4bef-be38-3dd6c7ba03a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:47.509347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.509327 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd" pod "3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" (UID: "3d2c7c60-8564-4bef-be38-3dd6c7ba03a4"). InnerVolumeSpecName "kube-api-access-hsfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:47.608270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.608245 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:47.608270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:47.608268 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsfbd\" (UniqueName: \"kubernetes.io/projected/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4-kube-api-access-hsfbd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:48.217581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:48.217553 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" Nov 03 20:25:48.217743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:48.217551 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd" event={"ID":"3d2c7c60-8564-4bef-be38-3dd6c7ba03a4","Type":"ContainerDied","Data":"c7e5c0e0d7cbd2cd54e9682bab88a3b3b64545f9edd2b5e3f0e77c8281663619"} Nov 03 20:25:48.217743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:48.217680 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e5c0e0d7cbd2cd54e9682bab88a3b3b64545f9edd2b5e3f0e77c8281663619" Nov 03 20:25:56.107681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:56.107634 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:56.252535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:56.227608 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.530533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:57.529593 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.639535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:57.611731 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.937635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:57.914628 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:25:58.537658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.537611 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:58.539979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.539956 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:25:58.697535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.689590 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:58.697535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.690167 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:25:58.873586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.843063 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:25:58.916852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.916809 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:58.962631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.958616 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:25:58.999616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:58.999561 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:25:59.151709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.151592 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:25:59.253169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.247929 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:59.427690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.427585 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5"] Nov 03 20:25:59.428865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.428843 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" containerName="revoke-keys" Nov 03 20:25:59.429016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.429003 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" containerName="revoke-keys" Nov 03 20:25:59.429167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.429155 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" Nov 03 20:25:59.429265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.429253 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" Nov 03 20:25:59.429574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.429558 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b63215f-9fd3-49d3-a6c5-c1104fe44376" containerName="main" Nov 03 20:25:59.429700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.429689 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" containerName="revoke-keys" Nov 03 20:25:59.467552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.461358 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.550535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.549047 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677028 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677106 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677202 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677238 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677295 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677327 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677392 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcrs\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677479 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.677532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.677523 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.790535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.782930 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.794700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.794667 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.798745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.798719 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.798982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.798965 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.799113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.799096 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.818846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.818815 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcrs\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.819139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.819119 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.819281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.819265 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.809083 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.820183 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.894321 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.786027 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.806374 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.841133 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.867097 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.805838 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:25:59.908536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.907357 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:00.002885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:25:59.960027 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:00.003240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.003165 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:00.029581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.029426 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcrs\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:00.031578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.031479 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5"] Nov 03 20:26:00.229516 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.224653 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:00.561801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.561772 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5"] Nov 03 20:26:00.563113 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:00.563087 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc257930_c536_4240_9e51_712cb9721b71.slice/crio-1ac604942b32cf0492a623d04b48a7150b37089aeee9cfb58bdff5dcae59cc67 WatchSource:0}: Error finding container 1ac604942b32cf0492a623d04b48a7150b37089aeee9cfb58bdff5dcae59cc67: Status 404 returned error can't find the container with id 1ac604942b32cf0492a623d04b48a7150b37089aeee9cfb58bdff5dcae59cc67 Nov 03 20:26:00.566198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.566161 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:00.566324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.566243 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:00.566324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:00.566283 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:01.393693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:01.392767 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" event={"ID":"bc257930-c536-4240-9e51-712cb9721b71","Type":"ContainerStarted","Data":"6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139"} Nov 03 20:26:01.393693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:01.392836 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" event={"ID":"bc257930-c536-4240-9e51-712cb9721b71","Type":"ContainerStarted","Data":"1ac604942b32cf0492a623d04b48a7150b37089aeee9cfb58bdff5dcae59cc67"} Nov 03 20:26:01.420025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:01.419967 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podStartSLOduration=2.4199481179999998 podStartE2EDuration="2.419948118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:01.418318408 +0000 UTC m=+1470.948664080" watchObservedRunningTime="2026-11-03 20:26:01.419948118 +0000 UTC m=+1470.950293791" Nov 03 20:26:02.225355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:02.225315 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:02.226751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:02.226723 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:26:02.226892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:02.226779 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 20:26:03.226528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:03.225938 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:26:03.226528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:03.226030 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 20:26:04.149389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.149357 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7"] Nov 03 20:26:04.154025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.154007 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.159585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.159484 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:04.159672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.159583 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:26:04.199025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.165065 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7"] Nov 03 20:26:04.225328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.225294 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:26:04.225555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.225351 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 20:26:04.235782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.235751 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.236145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.235984 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.236145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.236020 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqfr\" (UniqueName: \"kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.337657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.337625 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.337865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.337753 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.337865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.337777 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqfr\" (UniqueName: \"kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.338377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.338357 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.340156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.340131 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.350188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.350166 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqfr\" (UniqueName: \"kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr\") pod \"maas-api-revoke-sub-f86a8ac9-t7fk7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.526547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.524617 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:04.952288 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:04.952256 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951bf3c3_0618_41ed_a53d_485649fdc4f7.slice/crio-71f66bd04ac820b34d090f823d44d5ad62efb7d34dddcbbf7bdf7947f269d32b WatchSource:0}: Error finding container 71f66bd04ac820b34d090f823d44d5ad62efb7d34dddcbbf7bdf7947f269d32b: Status 404 returned error can't find the container with id 71f66bd04ac820b34d090f823d44d5ad62efb7d34dddcbbf7bdf7947f269d32b Nov 03 20:26:04.952576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:04.952555 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7"] Nov 03 20:26:05.232977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.225410 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:26:05.232977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.225477 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 20:26:05.517585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.509175 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" event={"ID":"951bf3c3-0618-41ed-a53d-485649fdc4f7","Type":"ContainerStarted","Data":"71f66bd04ac820b34d090f823d44d5ad62efb7d34dddcbbf7bdf7947f269d32b"} Nov 03 20:26:05.802926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.802845 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b"] Nov 03 20:26:05.811334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.811292 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:05.815920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.815893 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b"] Nov 03 20:26:05.958418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.956115 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:05.958418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.956160 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:05.958418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:05.956213 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp76x\" (UniqueName: \"kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.057307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.057227 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp76x\" (UniqueName: \"kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.057545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.057525 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.057610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.057584 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.058097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.058071 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.059994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.059975 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.106997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.106963 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp76x\" (UniqueName: \"kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x\") pod \"maas-api-revoke-sub-277b3255-zbl8b\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.117608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.110894 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6"] Nov 03 20:26:06.117608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.116286 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.122552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.122068 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6"] Nov 03 20:26:06.159384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.159359 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.159600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.159582 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdzz\" (UniqueName: \"kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.159661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.159618 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.226289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.226255 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:26:06.226433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.226314 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 20:26:06.253663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.253634 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:06.261270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.261241 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdzz\" (UniqueName: \"kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.261401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.261281 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.261494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.261394 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.262166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.262138 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.293690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.263905 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.305137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.305109 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdzz\" (UniqueName: \"kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz\") pod \"maas-api-revoke-sub-0d8c33cf-c8nb6\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.512209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.512175 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:06.520491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.520453 5815 generic.go:356] "Generic (PLEG): container finished" podID="951bf3c3-0618-41ed-a53d-485649fdc4f7" containerID="6e6231bc09714ff5e91a54f7dbab5302d09ccd5f62f25c793ce90a4491d9abe3" exitCode=0 Nov 03 20:26:06.520906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.520548 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" event={"ID":"951bf3c3-0618-41ed-a53d-485649fdc4f7","Type":"ContainerDied","Data":"6e6231bc09714ff5e91a54f7dbab5302d09ccd5f62f25c793ce90a4491d9abe3"} Nov 03 20:26:06.621801 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:06.621773 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26df59cc_659a_4c25_864c_7669e32e7593.slice/crio-41b5268a883eabc7559856b901acef012075d9ac7b7ca60c658845245fbc55c2 WatchSource:0}: Error finding container 41b5268a883eabc7559856b901acef012075d9ac7b7ca60c658845245fbc55c2: Status 404 returned error can't find the container with id 41b5268a883eabc7559856b901acef012075d9ac7b7ca60c658845245fbc55c2 Nov 03 20:26:06.638122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.638098 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b"] Nov 03 20:26:06.847918 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:06.847847 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d8e1d8_0361_4bd2_a93e_8fd079c3c470.slice/crio-0a4f26fccce13559857c6f5ef1f8850f83bbf1c6fea2ed02233933cf3706c63f WatchSource:0}: Error finding container 0a4f26fccce13559857c6f5ef1f8850f83bbf1c6fea2ed02233933cf3706c63f: Status 404 returned error can't find the container with id 0a4f26fccce13559857c6f5ef1f8850f83bbf1c6fea2ed02233933cf3706c63f Nov 03 20:26:06.851055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:06.851027 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6"] Nov 03 20:26:07.226077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.226047 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:26:07.226240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.226090 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 03 20:26:07.243930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.243895 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq"] Nov 03 20:26:07.248182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.248165 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.261298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.261276 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq"] Nov 03 20:26:07.330242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.330219 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.330452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.330430 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6lfh\" (UniqueName: \"kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.330499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.330471 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.431564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.431455 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lfh\" (UniqueName: \"kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.431564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.431496 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.431812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.431598 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.432252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.432228 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.434124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.434105 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.444112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.444084 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6lfh\" (UniqueName: \"kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh\") pod \"maas-api-revoke-sub-52586cb9-cjlfq\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.534550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.533417 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" event={"ID":"26df59cc-659a-4c25-864c-7669e32e7593","Type":"ContainerStarted","Data":"ba439f359fba4eaa8195a68096c45b54f83d7eca4079e32d9d02f31d06bdb5d4"} Nov 03 20:26:07.534550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.533495 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" event={"ID":"26df59cc-659a-4c25-864c-7669e32e7593","Type":"ContainerStarted","Data":"41b5268a883eabc7559856b901acef012075d9ac7b7ca60c658845245fbc55c2"} Nov 03 20:26:07.535566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.535519 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" event={"ID":"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470","Type":"ContainerStarted","Data":"2e09a00474448f4b6fc6a22946256aef20b195bbf8801a8800f082986bed9a52"} Nov 03 20:26:07.535728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.535591 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" event={"ID":"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470","Type":"ContainerStarted","Data":"0a4f26fccce13559857c6f5ef1f8850f83bbf1c6fea2ed02233933cf3706c63f"} Nov 03 20:26:07.553099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.553042 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" podStartSLOduration=2.553027157 podStartE2EDuration="2.553027157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:07.55199251 +0000 UTC m=+1477.082338183" watchObservedRunningTime="2026-11-03 20:26:07.553027157 +0000 UTC m=+1477.083372829" Nov 03 20:26:07.577644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.573734 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:07.577644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:07.574349 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" podStartSLOduration=1.574334026 podStartE2EDuration="1.574334026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:07.573557446 +0000 UTC m=+1477.103903112" watchObservedRunningTime="2026-11-03 20:26:07.574334026 +0000 UTC m=+1477.104679699" Nov 03 20:26:08.227820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.226214 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:08.319963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.318715 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca\") pod \"951bf3c3-0618-41ed-a53d-485649fdc4f7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " Nov 03 20:26:08.319963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.318845 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca\") pod \"951bf3c3-0618-41ed-a53d-485649fdc4f7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " Nov 03 20:26:08.319963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.319019 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr\" (UniqueName: \"kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr\") pod \"951bf3c3-0618-41ed-a53d-485649fdc4f7\" (UID: \"951bf3c3-0618-41ed-a53d-485649fdc4f7\") " Nov 03 20:26:08.320724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.320691 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca" pod "951bf3c3-0618-41ed-a53d-485649fdc4f7" (UID: "951bf3c3-0618-41ed-a53d-485649fdc4f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:08.324048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.324002 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca" pod "951bf3c3-0618-41ed-a53d-485649fdc4f7" (UID: "951bf3c3-0618-41ed-a53d-485649fdc4f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:08.324048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.324025 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr" pod "951bf3c3-0618-41ed-a53d-485649fdc4f7" (UID: "951bf3c3-0618-41ed-a53d-485649fdc4f7"). InnerVolumeSpecName "kube-api-access-4nqfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:08.371777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.371749 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq"] Nov 03 20:26:08.374684 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:08.374645 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cec863f_dd73_40d1_8b89_3f6fb872a81f.slice/crio-45345fcd2fd76accd46517f83ea679699831d2653960ef171e16995febe74ba9 WatchSource:0}: Error finding container 45345fcd2fd76accd46517f83ea679699831d2653960ef171e16995febe74ba9: Status 404 returned error can't find the container with id 45345fcd2fd76accd46517f83ea679699831d2653960ef171e16995febe74ba9 Nov 03 20:26:08.421549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.421304 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqfr\" (UniqueName: \"kubernetes.io/projected/951bf3c3-0618-41ed-a53d-485649fdc4f7-kube-api-access-4nqfr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:08.421549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.421339 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:08.421549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.421356 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951bf3c3-0618-41ed-a53d-485649fdc4f7-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:08.549180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.549140 5815 generic.go:356] "Generic (PLEG): container finished" podID="c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" containerID="2e09a00474448f4b6fc6a22946256aef20b195bbf8801a8800f082986bed9a52" exitCode=0 Nov 03 20:26:08.549673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.549240 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" event={"ID":"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470","Type":"ContainerDied","Data":"2e09a00474448f4b6fc6a22946256aef20b195bbf8801a8800f082986bed9a52"} Nov 03 20:26:08.551317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.551284 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" Nov 03 20:26:08.551317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.551303 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7" event={"ID":"951bf3c3-0618-41ed-a53d-485649fdc4f7","Type":"ContainerDied","Data":"71f66bd04ac820b34d090f823d44d5ad62efb7d34dddcbbf7bdf7947f269d32b"} Nov 03 20:26:08.551543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.551344 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f66bd04ac820b34d090f823d44d5ad62efb7d34dddcbbf7bdf7947f269d32b" Nov 03 20:26:08.560158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.556010 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" event={"ID":"8cec863f-dd73-40d1-8b89-3f6fb872a81f","Type":"ContainerStarted","Data":"45345fcd2fd76accd46517f83ea679699831d2653960ef171e16995febe74ba9"} Nov 03 20:26:08.560479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.560448 5815 generic.go:356] "Generic (PLEG): container finished" podID="26df59cc-659a-4c25-864c-7669e32e7593" containerID="ba439f359fba4eaa8195a68096c45b54f83d7eca4079e32d9d02f31d06bdb5d4" exitCode=0 Nov 03 20:26:08.560626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.560608 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" event={"ID":"26df59cc-659a-4c25-864c-7669e32e7593","Type":"ContainerDied","Data":"ba439f359fba4eaa8195a68096c45b54f83d7eca4079e32d9d02f31d06bdb5d4"} Nov 03 20:26:08.965653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.965626 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:08.967715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.967620 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:08.968788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:08.968767 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:26:09.572713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.572676 5815 generic.go:356] "Generic (PLEG): container finished" podID="8cec863f-dd73-40d1-8b89-3f6fb872a81f" containerID="22fb07553895321973c0acd8327c1405d087f2cef6bacc8d1e7e93e91bce91f3" exitCode=0 Nov 03 20:26:09.573204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.572792 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" event={"ID":"8cec863f-dd73-40d1-8b89-3f6fb872a81f","Type":"ContainerDied","Data":"22fb07553895321973c0acd8327c1405d087f2cef6bacc8d1e7e93e91bce91f3"} Nov 03 20:26:09.745021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.744992 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:09.779866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.779836 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz\" (UniqueName: \"kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz\") pod \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " Nov 03 20:26:09.780019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.779928 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca\") pod \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " Nov 03 20:26:09.780143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.780126 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca\") pod \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\" (UID: \"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470\") " Nov 03 20:26:09.780632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.780604 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca" pod "c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" (UID: "c1d8e1d8-0361-4bd2-a93e-8fd079c3c470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:09.782395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.782371 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz" pod "c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" (UID: "c1d8e1d8-0361-4bd2-a93e-8fd079c3c470"). InnerVolumeSpecName "kube-api-access-cfdzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:09.782871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.782847 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca" pod "c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" (UID: "c1d8e1d8-0361-4bd2-a93e-8fd079c3c470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:09.794554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.794497 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:09.881942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.881906 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x\" (UniqueName: \"kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x\") pod \"26df59cc-659a-4c25-864c-7669e32e7593\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " Nov 03 20:26:09.882089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882055 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca\") pod \"26df59cc-659a-4c25-864c-7669e32e7593\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " Nov 03 20:26:09.882153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882118 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca\") pod \"26df59cc-659a-4c25-864c-7669e32e7593\" (UID: \"26df59cc-659a-4c25-864c-7669e32e7593\") " Nov 03 20:26:09.882482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882449 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca" pod "26df59cc-659a-4c25-864c-7669e32e7593" (UID: "26df59cc-659a-4c25-864c-7669e32e7593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:09.882815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882761 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:09.882815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882782 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:09.882815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882797 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:09.882815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.882811 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdzz\" (UniqueName: \"kubernetes.io/projected/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470-kube-api-access-cfdzz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:09.884295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.884266 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca" pod "26df59cc-659a-4c25-864c-7669e32e7593" (UID: "26df59cc-659a-4c25-864c-7669e32e7593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:09.884295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.884274 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x" pod "26df59cc-659a-4c25-864c-7669e32e7593" (UID: "26df59cc-659a-4c25-864c-7669e32e7593"). InnerVolumeSpecName "kube-api-access-zp76x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:09.991337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.991273 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26df59cc-659a-4c25-864c-7669e32e7593-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:09.991976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:09.991950 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp76x\" (UniqueName: \"kubernetes.io/projected/26df59cc-659a-4c25-864c-7669e32e7593-kube-api-access-zp76x\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:10.585827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.585784 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" Nov 03 20:26:10.587827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.587800 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" Nov 03 20:26:10.595406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.595379 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6" event={"ID":"c1d8e1d8-0361-4bd2-a93e-8fd079c3c470","Type":"ContainerDied","Data":"0a4f26fccce13559857c6f5ef1f8850f83bbf1c6fea2ed02233933cf3706c63f"} Nov 03 20:26:10.595604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.595587 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a4f26fccce13559857c6f5ef1f8850f83bbf1c6fea2ed02233933cf3706c63f" Nov 03 20:26:10.595725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.595711 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b" event={"ID":"26df59cc-659a-4c25-864c-7669e32e7593","Type":"ContainerDied","Data":"41b5268a883eabc7559856b901acef012075d9ac7b7ca60c658845245fbc55c2"} Nov 03 20:26:10.595870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.595857 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b5268a883eabc7559856b901acef012075d9ac7b7ca60c658845245fbc55c2" Nov 03 20:26:10.781922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.781897 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:10.920574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.914090 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh\" (UniqueName: \"kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh\") pod \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " Nov 03 20:26:10.920574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.914173 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca\") pod \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " Nov 03 20:26:10.920574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.914314 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca\") pod \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\" (UID: \"8cec863f-dd73-40d1-8b89-3f6fb872a81f\") " Nov 03 20:26:10.930481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.928985 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca" pod "8cec863f-dd73-40d1-8b89-3f6fb872a81f" (UID: "8cec863f-dd73-40d1-8b89-3f6fb872a81f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:10.941919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.940003 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh" pod "8cec863f-dd73-40d1-8b89-3f6fb872a81f" (UID: "8cec863f-dd73-40d1-8b89-3f6fb872a81f"). InnerVolumeSpecName "kube-api-access-z6lfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:10.945430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:10.945396 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca" pod "8cec863f-dd73-40d1-8b89-3f6fb872a81f" (UID: "8cec863f-dd73-40d1-8b89-3f6fb872a81f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:11.016717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:11.016687 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6lfh\" (UniqueName: \"kubernetes.io/projected/8cec863f-dd73-40d1-8b89-3f6fb872a81f-kube-api-access-z6lfh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:11.016717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:11.016714 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:11.016915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:11.016727 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cec863f-dd73-40d1-8b89-3f6fb872a81f-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:11.600446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:11.600414 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" Nov 03 20:26:11.600938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:11.600417 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq" event={"ID":"8cec863f-dd73-40d1-8b89-3f6fb872a81f","Type":"ContainerDied","Data":"45345fcd2fd76accd46517f83ea679699831d2653960ef171e16995febe74ba9"} Nov 03 20:26:11.600938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:11.600550 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45345fcd2fd76accd46517f83ea679699831d2653960ef171e16995febe74ba9" Nov 03 20:26:12.417287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:12.417256 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:26:14.132552 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:26:14.131924 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:26:15.058690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:15.058621 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:26:15.178535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:15.154600 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.220759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:15.220696 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:26:15.566862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:15.566817 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.846586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:15.845615 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.985532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:15.985025 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.244534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.243893 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.245053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.244894 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:26:16.611669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.600387 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.793308 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.819525 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk"] Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820584 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820616 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820630 5815 cpu_manager.go:414] "containerMap: removing container" podUID="26df59cc-659a-4c25-864c-7669e32e7593" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820636 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26df59cc-659a-4c25-864c-7669e32e7593" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820661 5815 cpu_manager.go:414] "containerMap: removing container" podUID="951bf3c3-0618-41ed-a53d-485649fdc4f7" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820681 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951bf3c3-0618-41ed-a53d-485649fdc4f7" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820694 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8cec863f-dd73-40d1-8b89-3f6fb872a81f" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820700 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cec863f-dd73-40d1-8b89-3f6fb872a81f" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820904 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820917 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="26df59cc-659a-4c25-864c-7669e32e7593" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820927 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="951bf3c3-0618-41ed-a53d-485649fdc4f7" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.820937 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cec863f-dd73-40d1-8b89-3f6fb872a81f" containerName="revoke-keys" Nov 03 20:26:16.889660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:16.869137 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.014472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.012540 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.014472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.013152 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:26:17.016906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.016881 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3c25bf-openshift-default-dockercfg-bmp7d\"" Nov 03 20:26:17.115339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.109705 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223002 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223263 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223324 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223396 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223535 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkhrq\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223594 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223661 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.223702 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.114943 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:26:17.230643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.130634 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk"] Nov 03 20:26:17.269531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.252344 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.269531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.260906 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:26:17.269531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.260999 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.396607 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.396662 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.396842 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.396881 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.396921 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.396988 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkhrq\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.397039 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.403915 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.419674 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.424840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.420400 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.440549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.428777 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.440549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.437278 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.440952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.440896 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.444359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.441336 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.528487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.528409 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.540680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.537582 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.549597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.549562 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c"] Nov 03 20:26:17.604535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.582254 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkhrq\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.639901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.639868 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c"] Nov 03 20:26:17.649558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.646144 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.649558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.646715 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:17.789600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.786563 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:26:17.789600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.786696 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:26:17.814807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.807347 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:17.841902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.840947 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:26:17.875237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.875049 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.987177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.986468 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:26:17.998278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.993304 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:17.998278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.993376 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2r8\" (UniqueName: \"kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:17.998278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:17.993460 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.012537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.004782 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:18.108609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.104641 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.108609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.104721 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2r8\" (UniqueName: \"kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.108609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.104794 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.111597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.110262 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.277548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.240932 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.277548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.250334 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2r8\" (UniqueName: \"kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8\") pod \"maas-api-revoke-sub-df0e745a-h6k2c\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:18.316934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.316625 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:18.316934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.316683 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:18.418586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.413968 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:26:18.520532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:18.516342 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:19.236747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:19.236710 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk"] Nov 03 20:26:19.341069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:19.340478 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:19.341069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:19.340598 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:19.341069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:19.340645 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:19.612697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:19.612667 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c"] Nov 03 20:26:20.329791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:20.329749 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" event={"ID":"18812c7c-ac28-4334-a508-d73669a0cf0a","Type":"ContainerStarted","Data":"6e44bb7679d0b8ba9edc7294f5a60371bedc1fd2c8144cdade032ba28ed091b8"} Nov 03 20:26:20.329990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:20.329806 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" event={"ID":"18812c7c-ac28-4334-a508-d73669a0cf0a","Type":"ContainerStarted","Data":"d2670d078c6d1c3f356b3b81ad7cbb01087cf8de69e4f8d278c8fb6c16c78d1d"} Nov 03 20:26:20.331584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:20.331554 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" event={"ID":"6ca9be54-6a5f-43ca-93a5-c6666c72719a","Type":"ContainerStarted","Data":"d7ec305164ad933b663dab19d1f6a6eed267ba9c98db655cdb3a6debe83b7222"} Nov 03 20:26:20.331690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:20.331599 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" event={"ID":"6ca9be54-6a5f-43ca-93a5-c6666c72719a","Type":"ContainerStarted","Data":"c37958d8fc4b9159fde1b0242fabe5e86abdd3d38f53e85fac009593e39a871c"} Nov 03 20:26:20.374665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:20.374622 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podStartSLOduration=4.374609267 podStartE2EDuration="4.374609267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:20.372561038 +0000 UTC m=+1489.902906737" watchObservedRunningTime="2026-11-03 20:26:20.374609267 +0000 UTC m=+1489.904954939" Nov 03 20:26:20.394959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:20.394906 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" podStartSLOduration=3.39488919 podStartE2EDuration="3.39488919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:20.391229318 +0000 UTC m=+1489.921574992" watchObservedRunningTime="2026-11-03 20:26:20.39488919 +0000 UTC m=+1489.925234863" Nov 03 20:26:21.017741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.017704 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:21.022378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.022353 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:21.341148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.341065 5815 generic.go:356] "Generic (PLEG): container finished" podID="6ca9be54-6a5f-43ca-93a5-c6666c72719a" containerID="d7ec305164ad933b663dab19d1f6a6eed267ba9c98db655cdb3a6debe83b7222" exitCode=0 Nov 03 20:26:21.341330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.341166 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" event={"ID":"6ca9be54-6a5f-43ca-93a5-c6666c72719a","Type":"ContainerDied","Data":"d7ec305164ad933b663dab19d1f6a6eed267ba9c98db655cdb3a6debe83b7222"} Nov 03 20:26:21.341628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.341614 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:21.342424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.342408 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:26:21.651762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.651687 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp"] Nov 03 20:26:21.655866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.655847 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.665589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.665564 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp"] Nov 03 20:26:21.764808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.764779 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.764987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.764914 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mknpj\" (UniqueName: \"kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.764987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.764969 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.866196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.866165 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.866393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.866260 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mknpj\" (UniqueName: \"kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.866393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.866300 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.866954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.866934 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.869115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.869091 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.878766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.878731 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mknpj\" (UniqueName: \"kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj\") pod \"maas-api-revoke-sub-69319350-x5wtp\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:21.976268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:21.976238 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:22.267702 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:22.267669 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b4f970_620c_47f4_9b37_a65f230f7835.slice/crio-c355c865ea7276d5441212123d0579518b9d7deba4e326e35909d08b577f30dd WatchSource:0}: Error finding container c355c865ea7276d5441212123d0579518b9d7deba4e326e35909d08b577f30dd: Status 404 returned error can't find the container with id c355c865ea7276d5441212123d0579518b9d7deba4e326e35909d08b577f30dd Nov 03 20:26:22.268750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.268723 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp"] Nov 03 20:26:22.429528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.423796 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" event={"ID":"e5b4f970-620c-47f4-9b37-a65f230f7835","Type":"ContainerStarted","Data":"c355c865ea7276d5441212123d0579518b9d7deba4e326e35909d08b577f30dd"} Nov 03 20:26:22.593647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.593621 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:22.705825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.705786 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8\" (UniqueName: \"kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8\") pod \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " Nov 03 20:26:22.706237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.705891 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca\") pod \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " Nov 03 20:26:22.706237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.705943 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca\") pod \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\" (UID: \"6ca9be54-6a5f-43ca-93a5-c6666c72719a\") " Nov 03 20:26:22.706457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.706428 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca" pod "6ca9be54-6a5f-43ca-93a5-c6666c72719a" (UID: "6ca9be54-6a5f-43ca-93a5-c6666c72719a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:22.706969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.706936 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:22.708391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.708356 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca" pod "6ca9be54-6a5f-43ca-93a5-c6666c72719a" (UID: "6ca9be54-6a5f-43ca-93a5-c6666c72719a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:22.708493 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.708454 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8" pod "6ca9be54-6a5f-43ca-93a5-c6666c72719a" (UID: "6ca9be54-6a5f-43ca-93a5-c6666c72719a"). InnerVolumeSpecName "kube-api-access-xh2r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:22.808910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.808873 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca9be54-6a5f-43ca-93a5-c6666c72719a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:22.809057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:22.808912 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh2r8\" (UniqueName: \"kubernetes.io/projected/6ca9be54-6a5f-43ca-93a5-c6666c72719a-kube-api-access-xh2r8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.433581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:23.433470 5815 generic.go:356] "Generic (PLEG): container finished" podID="e5b4f970-620c-47f4-9b37-a65f230f7835" containerID="60c07de504fc37e73300da72f93baed9062433be048acef5a5479b42d87a6681" exitCode=0 Nov 03 20:26:23.433581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:23.433564 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" event={"ID":"e5b4f970-620c-47f4-9b37-a65f230f7835","Type":"ContainerDied","Data":"60c07de504fc37e73300da72f93baed9062433be048acef5a5479b42d87a6681"} Nov 03 20:26:23.435184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:23.435160 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" Nov 03 20:26:23.435322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:23.435201 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c" event={"ID":"6ca9be54-6a5f-43ca-93a5-c6666c72719a","Type":"ContainerDied","Data":"c37958d8fc4b9159fde1b0242fabe5e86abdd3d38f53e85fac009593e39a871c"} Nov 03 20:26:23.435322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:23.435263 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c37958d8fc4b9159fde1b0242fabe5e86abdd3d38f53e85fac009593e39a871c" Nov 03 20:26:24.649524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.649481 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:24.714581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.714551 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb"] Nov 03 20:26:24.715148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.715134 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e5b4f970-620c-47f4-9b37-a65f230f7835" containerName="revoke-keys" Nov 03 20:26:24.715148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.715148 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b4f970-620c-47f4-9b37-a65f230f7835" containerName="revoke-keys" Nov 03 20:26:24.715274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.715187 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6ca9be54-6a5f-43ca-93a5-c6666c72719a" containerName="revoke-keys" Nov 03 20:26:24.715274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.715192 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca9be54-6a5f-43ca-93a5-c6666c72719a" containerName="revoke-keys" Nov 03 20:26:24.715373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.715291 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca9be54-6a5f-43ca-93a5-c6666c72719a" containerName="revoke-keys" Nov 03 20:26:24.715373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.715303 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b4f970-620c-47f4-9b37-a65f230f7835" containerName="revoke-keys" Nov 03 20:26:24.719073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.719052 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.729079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.729058 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-08a7bc-openshift-default-dockercfg-gpxmk\"" Nov 03 20:26:24.754692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.754662 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb"] Nov 03 20:26:24.762012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.761970 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj\" (UniqueName: \"kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj\") pod \"e5b4f970-620c-47f4-9b37-a65f230f7835\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " Nov 03 20:26:24.762463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.762443 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca\") pod \"e5b4f970-620c-47f4-9b37-a65f230f7835\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " Nov 03 20:26:24.762622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.762607 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca\") pod \"e5b4f970-620c-47f4-9b37-a65f230f7835\" (UID: \"e5b4f970-620c-47f4-9b37-a65f230f7835\") " Nov 03 20:26:24.762980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.762959 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.763192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.763178 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.763359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.763344 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.763470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.763458 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.763633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.763619 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.763783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.763768 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2kd\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.763911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.763896 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.764048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.764036 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.764143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.764131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.771742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.766970 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca" pod "e5b4f970-620c-47f4-9b37-a65f230f7835" (UID: "e5b4f970-620c-47f4-9b37-a65f230f7835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:24.771742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.767811 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca" pod "e5b4f970-620c-47f4-9b37-a65f230f7835" (UID: "e5b4f970-620c-47f4-9b37-a65f230f7835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:24.773554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.772802 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj" pod "e5b4f970-620c-47f4-9b37-a65f230f7835" (UID: "e5b4f970-620c-47f4-9b37-a65f230f7835"). InnerVolumeSpecName "kube-api-access-mknpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:24.865599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.865547 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2kd\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.865802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.865642 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.865802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.865711 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.865802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.865739 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.865981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.865843 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.865990 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866058 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866091 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866134 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866174 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866273 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.866301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866289 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b4f970-620c-47f4-9b37-a65f230f7835-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.866460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866305 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mknpj\" (UniqueName: \"kubernetes.io/projected/e5b4f970-620c-47f4-9b37-a65f230f7835-kube-api-access-mknpj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:24.866460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866319 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866392 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.866807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.866783 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.868380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.868362 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.869210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.869187 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.881801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.881776 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2kd\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:24.881881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:24.881816 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token\") pod \"e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:25.049061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.048761 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:25.399610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.390973 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd"] Nov 03 20:26:25.399610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.391796 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" podUID="9cf95310-a3f1-4475-963e-a56d544d7ae3" containerName="payload-pre-processing" containerID="cri-o://7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518" gracePeriod=30 Nov 03 20:26:25.473942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.473911 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" Nov 03 20:26:25.474088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.474058 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp" event={"ID":"e5b4f970-620c-47f4-9b37-a65f230f7835","Type":"ContainerDied","Data":"c355c865ea7276d5441212123d0579518b9d7deba4e326e35909d08b577f30dd"} Nov 03 20:26:25.474154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.474126 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c355c865ea7276d5441212123d0579518b9d7deba4e326e35909d08b577f30dd" Nov 03 20:26:25.552781 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:26:25.552746 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf95310_a3f1_4475_963e_a56d544d7ae3.slice/crio-conmon-7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:26:25.755531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.745962 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" podUID="9cf95310-a3f1-4475-963e-a56d544d7ae3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.155:9004: connect: connection refused" Nov 03 20:26:25.951813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.951773 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb"] Nov 03 20:26:25.972556 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:25.970490 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d4bf87_dd67_4905_bf8f_bb093e231c4d.slice/crio-5c5986c65cd789ca1e18b1e540371fec087e6491c24d5e6d5db6d8a6d8a87f3b WatchSource:0}: Error finding container 5c5986c65cd789ca1e18b1e540371fec087e6491c24d5e6d5db6d8a6d8a87f3b: Status 404 returned error can't find the container with id 5c5986c65cd789ca1e18b1e540371fec087e6491c24d5e6d5db6d8a6d8a87f3b Nov 03 20:26:25.973599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.973553 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:25.973710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.973649 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:25.973710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:25.973693 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:26.087834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.087811 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:26:26.153302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.152118 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx\" (UniqueName: \"kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx\") pod \"9cf95310-a3f1-4475-963e-a56d544d7ae3\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " Nov 03 20:26:26.153302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.152190 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume\") pod \"9cf95310-a3f1-4475-963e-a56d544d7ae3\" (UID: \"9cf95310-a3f1-4475-963e-a56d544d7ae3\") " Nov 03 20:26:26.153302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.152643 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume" pod "9cf95310-a3f1-4475-963e-a56d544d7ae3" (UID: "9cf95310-a3f1-4475-963e-a56d544d7ae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:26.177527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.164489 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cf95310-a3f1-4475-963e-a56d544d7ae3-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:26.177527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.166861 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx" pod "9cf95310-a3f1-4475-963e-a56d544d7ae3" (UID: "9cf95310-a3f1-4475-963e-a56d544d7ae3"). InnerVolumeSpecName "kube-api-access-mbdvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:26.266202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.266172 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbdvx\" (UniqueName: \"kubernetes.io/projected/9cf95310-a3f1-4475-963e-a56d544d7ae3-kube-api-access-mbdvx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:26.488375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.488340 5815 generic.go:356] "Generic (PLEG): container finished" podID="9cf95310-a3f1-4475-963e-a56d544d7ae3" containerID="7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518" exitCode=0 Nov 03 20:26:26.488588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.488553 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" event={"ID":"9cf95310-a3f1-4475-963e-a56d544d7ae3","Type":"ContainerDied","Data":"7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518"} Nov 03 20:26:26.488664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.488564 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" Nov 03 20:26:26.488664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.488630 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd" event={"ID":"9cf95310-a3f1-4475-963e-a56d544d7ae3","Type":"ContainerDied","Data":"c4b8470a717a55e3edbacef8ada5043dbeda44e795197fc84dd26fd1516b2f80"} Nov 03 20:26:26.488770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.488663 5815 scope.go:122] "RemoveContainer" containerID="7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518" Nov 03 20:26:26.491447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.491417 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" event={"ID":"65d4bf87-dd67-4905-bf8f-bb093e231c4d","Type":"ContainerStarted","Data":"838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380"} Nov 03 20:26:26.491650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.491474 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" event={"ID":"65d4bf87-dd67-4905-bf8f-bb093e231c4d","Type":"ContainerStarted","Data":"5c5986c65cd789ca1e18b1e540371fec087e6491c24d5e6d5db6d8a6d8a87f3b"} Nov 03 20:26:26.504661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.504645 5815 scope.go:122] "RemoveContainer" containerID="7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518" Nov 03 20:26:26.505188 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:26:26.505127 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518\": container with ID starting with 7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518 not found: ID does not exist" containerID="7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518" Nov 03 20:26:26.505188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.505162 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518"} err="failed to get container status \"7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518\": rpc error: code = NotFound desc = could not find container \"7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518\": container with ID starting with 7dea9a49e5c53bdc62608995f0404e7adf15a1701f9d9d510c9d3e7f5fcbe518 not found: ID does not exist" Nov 03 20:26:26.520341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.520291 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podStartSLOduration=2.520274663 podStartE2EDuration="2.520274663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:26.517639289 +0000 UTC m=+1496.047984962" watchObservedRunningTime="2026-11-03 20:26:26.520274663 +0000 UTC m=+1496.050620337" Nov 03 20:26:26.536875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.536852 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd"] Nov 03 20:26:26.541627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.541606 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2eee8c-f9d7cc995-5gnjd"] Nov 03 20:26:26.588270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:26.588240 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf95310-a3f1-4475-963e-a56d544d7ae3" path="/var/lib/kubelet/pods/9cf95310-a3f1-4475-963e-a56d544d7ae3/volumes" Nov 03 20:26:27.050200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:27.050158 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:27.056367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:27.056345 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:27.502918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:27.502886 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:27.503831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:27.503810 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:26:28.456359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:28.456324 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck"] Nov 03 20:26:28.456943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:28.456875 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" containerID="cri-o://df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc" gracePeriod=30 Nov 03 20:26:32.875479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.875448 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq"] Nov 03 20:26:32.878013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.877969 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9cf95310-a3f1-4475-963e-a56d544d7ae3" containerName="payload-pre-processing" Nov 03 20:26:32.878235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.878216 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf95310-a3f1-4475-963e-a56d544d7ae3" containerName="payload-pre-processing" Nov 03 20:26:32.878602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.878571 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf95310-a3f1-4475-963e-a56d544d7ae3" containerName="payload-pre-processing" Nov 03 20:26:32.897531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.892652 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.915088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.915061 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bab2c3-openshift-default-dockercfg-bkx62\"" Nov 03 20:26:32.917883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.917858 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq"] Nov 03 20:26:32.955310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955275 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955395 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhrp\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955480 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955566 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955593 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955639 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955675 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955732 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:32.955853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:32.955779 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.056757 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.056830 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.056862 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.056901 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.056936 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.056988 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.057031 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.057118 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.057218 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhrp\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.057959 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.058354 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.058627 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.060927 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.061603 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.064501 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.083950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.083332 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.122769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.101321 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhrp\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.122769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.106649 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:33.251542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:33.248789 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:34.311211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.264792 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq"] Nov 03 20:26:34.407535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.402738 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:34.407535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.402947 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:34.407535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.403131 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:34.407535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.403652 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:26:34.611217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611158 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611244 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611326 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611374 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611432 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611467 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611751 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.611874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.611861 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert\") pod \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\" (UID: \"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc\") " Nov 03 20:26:34.612098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.612069 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:34.612371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.612349 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:34.612766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.612741 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:34.612896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.612874 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.612971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.612899 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.613113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.613094 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:34.613311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.613294 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:34.617724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.617694 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:34.620238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.620096 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:34.621636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.621608 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:34.622014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.621990 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t" pod "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" (UID: "4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc"). InnerVolumeSpecName "kube-api-access-76j7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:34.628963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.628936 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" event={"ID":"7aa8313c-5794-49fd-8e97-caae0f755923","Type":"ContainerStarted","Data":"2422bffea716c64fe7c1a026ed9b3482e14b9d01483a22b7631ed9dfc8f4eec3"} Nov 03 20:26:34.631142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.631116 5815 generic.go:356] "Generic (PLEG): container finished" podID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerID="df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc" exitCode=0 Nov 03 20:26:34.631253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.631234 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" event={"ID":"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc","Type":"ContainerDied","Data":"df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc"} Nov 03 20:26:34.631321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.631273 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" event={"ID":"4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc","Type":"ContainerDied","Data":"e1d7e79086e72d0c28f0ae026278d66d6de91d7dd4aaf4e57b3c63fbe6e6f6d5"} Nov 03 20:26:34.631385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.631331 5815 scope.go:122] "RemoveContainer" containerID="df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc" Nov 03 20:26:34.631498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.631484 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck" Nov 03 20:26:34.647812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.647794 5815 scope.go:122] "RemoveContainer" containerID="df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc" Nov 03 20:26:34.648715 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:26:34.648689 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc\": container with ID starting with df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc not found: ID does not exist" containerID="df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc" Nov 03 20:26:34.648792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.648729 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc"} err="failed to get container status \"df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc\": rpc error: code = NotFound desc = could not find container \"df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc\": container with ID starting with df8dfa461386b4b94bdac84e90f2b4bd9b2766021def31abbaf55617d776facc not found: ID does not exist" Nov 03 20:26:34.716589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.714949 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.721958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.721915 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76j7t\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-kube-api-access-76j7t\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.722174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.722159 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.722267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.722255 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.722356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.722345 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.722436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.722426 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.722552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.722537 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:34.722966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.722944 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck"] Nov 03 20:26:34.729801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.729774 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-984ck"] Nov 03 20:26:34.791901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.791874 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:26:34.807195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.807119 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:26:34.818479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.818455 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:26:34.833429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:34.830846 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:26:35.642104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:35.642065 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" event={"ID":"7aa8313c-5794-49fd-8e97-caae0f755923","Type":"ContainerStarted","Data":"0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4"} Nov 03 20:26:35.671944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:35.671898 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podStartSLOduration=3.671884179 podStartE2EDuration="3.671884179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:35.668252208 +0000 UTC m=+1505.198597881" watchObservedRunningTime="2026-11-03 20:26:35.671884179 +0000 UTC m=+1505.202229882" Nov 03 20:26:36.250021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:36.249993 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:36.255034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:36.255010 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:36.587292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:36.587220 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" path="/var/lib/kubelet/pods/4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc/volumes" Nov 03 20:26:36.652431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:36.652401 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:36.653419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:36.653399 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:26:38.354966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:38.348347 5815 scope.go:122] "RemoveContainer" containerID="19fc447ea48a16ad06e3aac667aaacd0aedd4b4bdfac28cb49772c80280abfb1" Nov 03 20:26:39.264552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.264521 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd"] Nov 03 20:26:39.265109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.265096 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" Nov 03 20:26:39.265160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.265110 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" Nov 03 20:26:39.265231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.265222 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec14e6b-b1eb-46b8-bee7-1eef6138bdfc" containerName="istio-proxy" Nov 03 20:26:39.271039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.270560 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.277310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.276669 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd"] Nov 03 20:26:39.347974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.347845 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bab2c3-dockercfg-tjg8h\"" Nov 03 20:26:39.347974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.347943 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bab2c3\"" Nov 03 20:26:39.404533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.394770 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.404533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.394852 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcphd\" (UniqueName: \"kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd\") pod \"payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.536590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.496149 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.536938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.536909 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcphd\" (UniqueName: \"kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd\") pod \"payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.573038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.551239 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.573038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.572658 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcphd\" (UniqueName: \"kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd\") pod \"payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.611534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.606785 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:39.958498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:39.958469 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd"] Nov 03 20:26:40.729435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:40.729403 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" event={"ID":"f7cb8fbc-4f0b-4288-a297-f8ce9d066613","Type":"ContainerStarted","Data":"8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293"} Nov 03 20:26:40.729955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:40.729460 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" event={"ID":"f7cb8fbc-4f0b-4288-a297-f8ce9d066613","Type":"ContainerStarted","Data":"bda263136d6d5b1ea5c7a7fb0919d7ad3b401cc0f24812d157f0fd73ad48a012"} Nov 03 20:26:40.729955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:40.729494 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:40.753574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:40.753520 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" podStartSLOduration=1.753485491 podStartE2EDuration="1.753485491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:40.74826962 +0000 UTC m=+1510.278615294" watchObservedRunningTime="2026-11-03 20:26:40.753485491 +0000 UTC m=+1510.283831164" Nov 03 20:26:43.711534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:43.697645 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:43.840788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:43.835929 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:26:43.899535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:43.898636 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.001700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:44.001591 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.152650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:44.152600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.166240 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:26:44.166207 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:26:44.704657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:44.704600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.705524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:44.705483 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:26:44.921542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:44.905232 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:26:44.921542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:44.905694 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:26:45.316543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:45.316463 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 03 20:26:45.316747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:45.316606 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:26:45.324539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:45.317076 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:26:45.909658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:45.909611 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:45.952533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:45.923288 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:46.173754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.001604 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:46.175471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.011290 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm"] Nov 03 20:26:46.175897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.175867 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" containerID="cri-o://ebff6c5a2722ad0fe8b6a6d923785cf8ccc34dba8704497f52c046ce0d8003a2" gracePeriod=30 Nov 03 20:26:46.176141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.050335 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:26:46.224682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.224654 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:26:46.329207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.329154 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:46.353696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.339725 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:26:46.910856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:46.910803 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:47.138653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.138624 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:26:47.252535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.251111 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:47.334622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.334580 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:26:47.335389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.335364 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:26:47.472092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.472065 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:26:47.576137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.576014 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:26:47.751679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.751641 5815 patch_prober.go:29] interesting pod/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:26:47.751998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.751967 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:26:47.965132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:47.964043 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8"] Nov 03 20:26:48.109530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.102842 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.117669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.117642 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3211a0-openshift-default-dockercfg-57hrh\"" Nov 03 20:26:48.141107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.141023 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8"] Nov 03 20:26:48.323334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.323229 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.323641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.323620 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.323846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.323814 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.323998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.323980 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.324161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.324135 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsj5v\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.324411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.324381 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.324745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.324715 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.324953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.324923 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.325097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.325082 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.408915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.408857 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:26:48.409292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.409188 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:48.505288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.505242 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.505701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.505676 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.506004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.505985 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.506155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.506141 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.506325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.506311 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.506444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.506429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.506633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.506616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsj5v\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.506890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.506873 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.507139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.507122 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.507702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.507682 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.508078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.508059 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.508629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.508610 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.509262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.509240 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.536044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.520172 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.577808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.577693 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:48.596038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.596002 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.621102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.621036 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.643377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.643342 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.669818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.666163 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsj5v\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:48.741997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:48.741967 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:49.262409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.262384 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8"] Nov 03 20:26:49.264663 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:49.264631 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406afe59_8851_4178_8e7e_fad68db839ad.slice/crio-f94443d65573b9e88dce4d61dd08e96d3c808eeae3d068106f25e6ef4dc47c87 WatchSource:0}: Error finding container f94443d65573b9e88dce4d61dd08e96d3c808eeae3d068106f25e6ef4dc47c87: Status 404 returned error can't find the container with id f94443d65573b9e88dce4d61dd08e96d3c808eeae3d068106f25e6ef4dc47c87 Nov 03 20:26:49.267611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.267570 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:49.267697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.267658 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:49.267739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.267699 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:49.577719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.576948 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2"] Nov 03 20:26:49.614958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.614933 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.616172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.616131 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" event={"ID":"406afe59-8851-4178-8e7e-fad68db839ad","Type":"ContainerStarted","Data":"441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183"} Nov 03 20:26:49.617105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.617083 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" event={"ID":"406afe59-8851-4178-8e7e-fad68db839ad","Type":"ContainerStarted","Data":"f94443d65573b9e88dce4d61dd08e96d3c808eeae3d068106f25e6ef4dc47c87"} Nov 03 20:26:49.617202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.617120 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2"] Nov 03 20:26:49.620536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.620492 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-30e9de-openshift-default-dockercfg-j5fsc\"" Nov 03 20:26:49.641929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.641868 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.641929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.641914 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642062 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642109 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642267 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642401 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642445 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642540 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4x9g\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.642605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.642581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.647336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.647294 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podStartSLOduration=2.647279583 podStartE2EDuration="2.647279583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:49.644869947 +0000 UTC m=+1519.175215655" watchObservedRunningTime="2026-11-03 20:26:49.647279583 +0000 UTC m=+1519.177625255" Nov 03 20:26:49.744328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744294 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.744496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744368 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.744496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744431 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.744496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744455 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.744691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744502 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4x9g\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.744691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744567 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.744956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744925 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.744971 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.745066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.745092 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.745382 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.745562 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.745607 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.745686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.745678 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.747318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.747290 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:49.747592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.747573 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.747832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.747815 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.757415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.757393 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4x9g\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.757607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.757400 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:49.949077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:49.948770 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:50.464838 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:50.464786 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5ecf4d_7071_4b55_a6e2_de1108ce8ff0.slice/crio-9808decf9ca928cb1ee666275f2278b1732489d9df5decffd2cbfab876c65aa5 WatchSource:0}: Error finding container 9808decf9ca928cb1ee666275f2278b1732489d9df5decffd2cbfab876c65aa5: Status 404 returned error can't find the container with id 9808decf9ca928cb1ee666275f2278b1732489d9df5decffd2cbfab876c65aa5 Nov 03 20:26:50.469044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:50.469014 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2"] Nov 03 20:26:50.471409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:50.471364 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:50.471526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:50.471450 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:50.471526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:50.471496 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:26:50.643571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:50.643540 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" event={"ID":"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0","Type":"ContainerStarted","Data":"9808decf9ca928cb1ee666275f2278b1732489d9df5decffd2cbfab876c65aa5"} Nov 03 20:26:50.755940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:50.755855 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:51.661894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.657701 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" event={"ID":"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0","Type":"ContainerStarted","Data":"a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246"} Nov 03 20:26:51.661894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.660644 5815 generic.go:356] "Generic (PLEG): container finished" podID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerID="ebff6c5a2722ad0fe8b6a6d923785cf8ccc34dba8704497f52c046ce0d8003a2" exitCode=0 Nov 03 20:26:51.661894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.661829 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" event={"ID":"9b80b43d-1822-4c79-8a0f-bc60123fdf41","Type":"ContainerDied","Data":"ebff6c5a2722ad0fe8b6a6d923785cf8ccc34dba8704497f52c046ce0d8003a2"} Nov 03 20:26:51.661894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.661882 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:51.662670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.661913 5815 scope.go:122] "RemoveContainer" containerID="03369573c818494f7d5d3c6e9d3094d9e1d5bcac8262f3581e5c8a6535864e7b" Nov 03 20:26:51.663018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.662996 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:26:51.715994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.715933 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podStartSLOduration=2.715915109 podStartE2EDuration="2.715915109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:51.70559965 +0000 UTC m=+1521.235945326" watchObservedRunningTime="2026-11-03 20:26:51.715915109 +0000 UTC m=+1521.246260784" Nov 03 20:26:51.728974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.728950 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:26:51.742785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.742764 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 20:26:51.842132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842035 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842122 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842163 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842265 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842324 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842405 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842607 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842677 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs\") pod \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\" (UID: \"9b80b43d-1822-4c79-8a0f-bc60123fdf41\") " Nov 03 20:26:51.842909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.842880 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.843114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.843076 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.844017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.843294 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.844017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.843691 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:51.844155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.844090 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.844155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.844103 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.844155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.844129 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.844155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.844144 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.849068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.849030 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:51.855501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.855460 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "kube-api-access-7rs6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:51.858731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.857199 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:51.859434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.859409 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo" pod "9b80b43d-1822-4c79-8a0f-bc60123fdf41" (UID: "9b80b43d-1822-4c79-8a0f-bc60123fdf41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:51.945806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.945774 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rs6w\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-kube-api-access-7rs6w\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.945806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.945801 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.945806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.945810 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.946110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.945819 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.946110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.945827 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.946110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.945837 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b80b43d-1822-4c79-8a0f-bc60123fdf41-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:51.950069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.950039 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:51.955955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:51.955933 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:52.705038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.705012 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" Nov 03 20:26:52.710804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.710769 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm" event={"ID":"9b80b43d-1822-4c79-8a0f-bc60123fdf41","Type":"ContainerDied","Data":"8b9c8a76f3bd0583e5d94ee6edb0f010f12acf5d6dd730c3d38d69c53454edac"} Nov 03 20:26:52.711038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.711023 5815 scope.go:122] "RemoveContainer" containerID="ebff6c5a2722ad0fe8b6a6d923785cf8ccc34dba8704497f52c046ce0d8003a2" Nov 03 20:26:52.720264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.720192 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:52.727261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.727179 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:26:52.802882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.802850 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm"] Nov 03 20:26:52.949876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:52.949750 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1142fd-openshift-default-5d4d9469c-r7szm"] Nov 03 20:26:53.139159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.139123 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb"] Nov 03 20:26:53.140158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.140132 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" Nov 03 20:26:53.140158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.140157 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" Nov 03 20:26:53.140370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.140174 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" Nov 03 20:26:53.140370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.140182 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" Nov 03 20:26:53.140475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.140386 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" Nov 03 20:26:53.146000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.145959 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.157969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.157945 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3211a0\"" Nov 03 20:26:53.158119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.157967 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3211a0-dockercfg-kzdtp\"" Nov 03 20:26:53.158381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.158362 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb"] Nov 03 20:26:53.280673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.280637 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.281007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.280982 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p99t\" (UniqueName: \"kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t\") pod \"payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.386529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.383717 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.386529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.383786 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p99t\" (UniqueName: \"kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t\") pod \"payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.386529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.385006 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.465533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.463401 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p99t\" (UniqueName: \"kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t\") pod \"payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:53.482577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:53.479935 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:54.104862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:54.104791 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb"] Nov 03 20:26:54.595703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:54.595668 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" path="/var/lib/kubelet/pods/9b80b43d-1822-4c79-8a0f-bc60123fdf41/volumes" Nov 03 20:26:54.890633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:54.890542 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" event={"ID":"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d","Type":"ContainerStarted","Data":"6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626"} Nov 03 20:26:54.890633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:54.890597 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" event={"ID":"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d","Type":"ContainerStarted","Data":"6102df2be3a6200c44452ecc69741693684106b407c6904cbb4970177a0f4c6f"} Nov 03 20:26:54.890901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:54.890883 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:26:54.910426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:54.910375 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" podStartSLOduration=1.910361741 podStartE2EDuration="1.910361741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:54.909583798 +0000 UTC m=+1524.439929475" watchObservedRunningTime="2026-11-03 20:26:54.910361741 +0000 UTC m=+1524.440707412" Nov 03 20:26:55.586422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.586390 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn"] Nov 03 20:26:55.588526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.588485 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b80b43d-1822-4c79-8a0f-bc60123fdf41" containerName="istio-proxy" Nov 03 20:26:55.594572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.594550 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:55.607690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.607666 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn"] Nov 03 20:26:55.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.680719 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-30e9de-dockercfg-m6gp5\"" Nov 03 20:26:55.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.680833 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-30e9de\"" Nov 03 20:26:55.824599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.824557 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfnn\" (UniqueName: \"kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn\") pod \"payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:55.824983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.824959 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:55.926986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.926897 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfnn\" (UniqueName: \"kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn\") pod \"payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:55.927664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.927636 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:55.929033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.928769 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:55.984539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:55.979546 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfnn\" (UniqueName: \"kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn\") pod \"payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:56.216004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:56.215972 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:56.583269 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:26:56.583240 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cece6e8_e19d_4b4c_b121_43e69355418d.slice/crio-09b9ad46d0569fc57abead6b3dcc739edb99e8a53b7ef2a7f15e2ef01992479a WatchSource:0}: Error finding container 09b9ad46d0569fc57abead6b3dcc739edb99e8a53b7ef2a7f15e2ef01992479a: Status 404 returned error can't find the container with id 09b9ad46d0569fc57abead6b3dcc739edb99e8a53b7ef2a7f15e2ef01992479a Nov 03 20:26:56.591399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:56.591379 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn"] Nov 03 20:26:57.001678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:57.001637 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" event={"ID":"1cece6e8-e19d-4b4c-b121-43e69355418d","Type":"ContainerStarted","Data":"556a9d5789fbbf72315e6d4f8cd483f59fe4fb21f3feeedac78389afd2f4e220"} Nov 03 20:26:57.002083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:57.002062 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" event={"ID":"1cece6e8-e19d-4b4c-b121-43e69355418d","Type":"ContainerStarted","Data":"09b9ad46d0569fc57abead6b3dcc739edb99e8a53b7ef2a7f15e2ef01992479a"} Nov 03 20:26:57.006429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:57.006396 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:26:57.056941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:26:57.056829 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" podStartSLOduration=2.056808061 podStartE2EDuration="2.056808061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:57.046856034 +0000 UTC m=+1526.577201707" watchObservedRunningTime="2026-11-03 20:26:57.056808061 +0000 UTC m=+1526.587153776" Nov 03 20:27:00.789847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.789816 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v"] Nov 03 20:27:00.819544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.817721 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v"] Nov 03 20:27:00.819544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.817829 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:00.824145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.824067 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-7a92d4\"/\"openshift-service-ca.crt\"" Nov 03 20:27:00.824751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.824464 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-7a92d4\"/\"default-dockercfg-lmz7d\"" Nov 03 20:27:00.824903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.824495 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-7a92d4\"/\"auth-test-model-7a92d4-kserve-self-signed-certs\"" Nov 03 20:27:00.824903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.824615 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-7a92d4\"/\"kube-root-ca.crt\"" Nov 03 20:27:00.986532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.986159 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:00.986532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.986384 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:00.986532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.986426 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:00.986532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.986494 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:00.986915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.986581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:00.986915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:00.986609 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw5c\" (UniqueName: \"kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.096332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.096247 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.096332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.096305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.096675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.096373 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.096675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.096429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.096675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.096454 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw5c\" (UniqueName: \"kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.096675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.096562 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.097117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.097092 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.098209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.098147 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.098760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.098734 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.120153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.120110 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.142708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.140317 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw5c\" (UniqueName: \"kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.169178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.162906 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm\") pod \"auth-test-model-7a92d4-kserve-65f7db5545-4955v\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.436806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.436710 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:01.724627 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:27:01.724592 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2592b7c8_8386_4df0_8509_caf805154d4c.slice/crio-436ecd781124684cf8cd42a3fa861f9cd762a20268e9a0175e0ba915e4c63e8d WatchSource:0}: Error finding container 436ecd781124684cf8cd42a3fa861f9cd762a20268e9a0175e0ba915e4c63e8d: Status 404 returned error can't find the container with id 436ecd781124684cf8cd42a3fa861f9cd762a20268e9a0175e0ba915e4c63e8d Nov 03 20:27:01.725395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:01.725372 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v"] Nov 03 20:27:02.154817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:02.154721 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" event={"ID":"2592b7c8-8386-4df0-8509-caf805154d4c","Type":"ContainerStarted","Data":"01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186"} Nov 03 20:27:02.154817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:02.154763 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" event={"ID":"2592b7c8-8386-4df0-8509-caf805154d4c","Type":"ContainerStarted","Data":"436ecd781124684cf8cd42a3fa861f9cd762a20268e9a0175e0ba915e4c63e8d"} Nov 03 20:27:02.177353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:02.177294 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podStartSLOduration=2.177275453 podStartE2EDuration="2.177275453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:02.173258041 +0000 UTC m=+1531.703603712" watchObservedRunningTime="2026-11-03 20:27:02.177275453 +0000 UTC m=+1531.707621127" Nov 03 20:27:02.449578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:02.437605 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:04.105558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.105521 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp"] Nov 03 20:27:04.111454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.111425 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.126380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.126341 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp"] Nov 03 20:27:04.190853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.190818 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-99a8fe-openshift-default-dockercfg-5wmrx\"" Nov 03 20:27:04.260535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.252398 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.260919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.260889 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6gs\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.261080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.261063 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.265608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.265574 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.265815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.265794 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.303533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.302663 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.303533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.302768 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.303533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.302812 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.303533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.302942 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414424 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414542 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414587 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414694 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414861 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414903 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6gs\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.414952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414929 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.415043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.414964 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.415043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.415012 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.426549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.419886 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.426549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.420971 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.426549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.421955 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.426549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.422544 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.426549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.422770 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.449582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.445824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.472032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.453762 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.472032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.457405 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6gs\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.472932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.472866 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:04.752484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:04.752449 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:05.064436 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:27:05.064404 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ba818e_adb1_4432_ac0a_ca912d3c28c1.slice/crio-4392213ec9cc5f9bb8ee45fd9534674e9a5a931e6200fb3a86189aaac5d022e8 WatchSource:0}: Error finding container 4392213ec9cc5f9bb8ee45fd9534674e9a5a931e6200fb3a86189aaac5d022e8: Status 404 returned error can't find the container with id 4392213ec9cc5f9bb8ee45fd9534674e9a5a931e6200fb3a86189aaac5d022e8 Nov 03 20:27:05.065186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.065166 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp"] Nov 03 20:27:05.067565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.067500 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:05.067669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.067599 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:05.067669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.067630 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:05.187370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.187324 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" event={"ID":"b5ba818e-adb1-4432-ac0a-ca912d3c28c1","Type":"ContainerStarted","Data":"97d792a1ff7646d96b29370c2caf519086eee8a8794b18c64b119a79262dcc73"} Nov 03 20:27:05.187880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.187397 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" event={"ID":"b5ba818e-adb1-4432-ac0a-ca912d3c28c1","Type":"ContainerStarted","Data":"4392213ec9cc5f9bb8ee45fd9534674e9a5a931e6200fb3a86189aaac5d022e8"} Nov 03 20:27:05.216015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.212652 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podStartSLOduration=1.212633087 podStartE2EDuration="1.212633087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:05.209973408 +0000 UTC m=+1534.740319083" watchObservedRunningTime="2026-11-03 20:27:05.212633087 +0000 UTC m=+1534.742978762" Nov 03 20:27:05.754530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.752639 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:05.990577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:05.989300 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:27:06.760785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:06.760716 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:07.207833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:07.207752 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:07.208656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:07.208631 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:27:08.059389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:08.059362 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:27:08.909626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:08.909407 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx"] Nov 03 20:27:08.950953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:08.950925 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx"] Nov 03 20:27:08.951183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:08.951066 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:08.958526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:08.958479 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6e5139-openshift-default-dockercfg-7m2n7\"" Nov 03 20:27:09.133027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.132987 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.158992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.158952 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.172576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.159372 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.172974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.172950 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.173246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.173183 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.173405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.173387 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cx2\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.173669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.173651 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.173918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.173891 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.174122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.174104 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.298076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.298042 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.298468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.298432 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.298655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.298638 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.298780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.298767 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.298885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.298872 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cx2\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.298999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.298986 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.299106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.299093 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.299222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.299208 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.299593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.299573 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.300452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.300235 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.301109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.301091 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.302215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.302194 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.302856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.302834 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.315730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.315706 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.334555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.334214 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.336445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.335593 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.337090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.337066 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.346539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.345093 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cx2\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.581534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.581190 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:09.890053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.888923 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx"] Nov 03 20:27:09.897983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.897745 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:09.897983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.897871 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:09.899382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:09.897917 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:10.294885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:10.294832 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" event={"ID":"4fb00ef0-8914-4629-a6fe-bf7eee0b3688","Type":"ContainerStarted","Data":"b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b"} Nov 03 20:27:10.295414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:10.295390 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" event={"ID":"4fb00ef0-8914-4629-a6fe-bf7eee0b3688","Type":"ContainerStarted","Data":"143665b9dbbbae16a01caa108aba07c22e9476930f412600bf8cef5d82a4e05f"} Nov 03 20:27:10.320684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:10.320633 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podStartSLOduration=2.320619546 podStartE2EDuration="2.320619546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:10.319456161 +0000 UTC m=+1539.849801835" watchObservedRunningTime="2026-11-03 20:27:10.320619546 +0000 UTC m=+1539.850965218" Nov 03 20:27:10.623577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:10.617781 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:10.631820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:10.631789 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:11.310531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:11.309737 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:11.311030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:11.310965 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:27:11.437078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:11.437042 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:11.457993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:11.457963 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:12.329998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:12.329974 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:12.857551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:12.857521 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:27:13.854517 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:27:13.854460 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:27:16.019532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:15.997607 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:16.233533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:16.230731 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:16.325589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:16.324659 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:16.845530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:16.844604 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:17.915660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:17.915606 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:18.002388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:18.002342 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:18.323644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:18.323595 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:18.849541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:18.845691 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:20.937532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:20.931736 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:20.995655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:20.995612 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:21.086543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.084878 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787"] Nov 03 20:27:21.160533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.159983 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.179600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.168797 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-08a7bc\"/\"openshift-service-ca.crt\"" Nov 03 20:27:21.179600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.171747 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-08a7bc\"/\"auth-test-model-08a7bc-kserve-self-signed-certs\"" Nov 03 20:27:21.179600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.171827 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-08a7bc\"/\"default-dockercfg-nvm4v\"" Nov 03 20:27:21.179600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.171884 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-08a7bc\"/\"kube-root-ca.crt\"" Nov 03 20:27:21.210743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.210649 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787"] Nov 03 20:27:21.261531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.236985 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:21.351657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.342084 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.351657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.342131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.351657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.342173 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.351657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.342334 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmwx\" (UniqueName: \"kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.351657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.342447 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.351657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.342529 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.446375 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.446489 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.446538 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.446572 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.446707 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmwx\" (UniqueName: \"kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.446803 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.447249 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.447466 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.459045 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.459429 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.470582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.461967 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.498536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.497540 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmwx\" (UniqueName: \"kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx\") pod \"auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:21.558868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:21.558599 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:23.062193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:23.062150 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787"] Nov 03 20:27:23.239593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:23.238085 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" event={"ID":"f8df9da6-4c0d-43c4-9656-2b64fc448deb","Type":"ContainerStarted","Data":"77a83550c73d09c07ee2a77b1cb5ddd44d5161574ff13f6f4555cdf39ea7d3cd"} Nov 03 20:27:23.357539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:23.355012 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:24.869538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:24.861210 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:27:24.893990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:24.893952 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" event={"ID":"f8df9da6-4c0d-43c4-9656-2b64fc448deb","Type":"ContainerStarted","Data":"11170986b9a4ce745ae075f55cff68f3e9b9199eb158a4c51c1619d19aa9665d"} Nov 03 20:27:24.916622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:24.916581 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:24.986187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:24.986106 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podStartSLOduration=4.986083017 podStartE2EDuration="4.986083017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:24.943838872 +0000 UTC m=+1554.474184546" watchObservedRunningTime="2026-11-03 20:27:24.986083017 +0000 UTC m=+1554.516428691" Nov 03 20:27:25.010535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:25.006699 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:25.397635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:25.397590 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:25.399957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:25.399923 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:25.560255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:25.560180 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:31.560159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.559486 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.573055 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.620688 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74"] Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.640886 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74"] Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.641031 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.793719 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.793775 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.793924 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.793983 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.794093 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.794128 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.794255 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkpv\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.794287 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.856726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.794372 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896195 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896256 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896316 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896357 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896443 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkpv\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896473 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896546 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896598 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896674 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896723 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.896909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896841 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.897002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.896983 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.897047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.897013 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.897415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.897343 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.899130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.899094 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.899443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.899418 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.911043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.911015 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:31.911178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:31.911157 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkpv\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:32.040726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.040690 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:32.088741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.088704 5815 generic.go:356] "Generic (PLEG): container finished" podID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerID="6e44bb7679d0b8ba9edc7294f5a60371bedc1fd2c8144cdade032ba28ed091b8" exitCode=0 Nov 03 20:27:32.088876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.088779 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" event={"ID":"18812c7c-ac28-4334-a508-d73669a0cf0a","Type":"ContainerDied","Data":"6e44bb7679d0b8ba9edc7294f5a60371bedc1fd2c8144cdade032ba28ed091b8"} Nov 03 20:27:32.089754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.089716 5815 scope.go:122] "RemoveContainer" containerID="6e44bb7679d0b8ba9edc7294f5a60371bedc1fd2c8144cdade032ba28ed091b8" Nov 03 20:27:32.092067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.092032 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:32.092192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.092123 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:32.092192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.092169 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:32.109718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.109611 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:32.439178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.439054 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74"] Nov 03 20:27:32.482746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.479979 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:32.482746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.480067 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:32.482746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:32.480113 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.107541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.106321 5815 generic.go:356] "Generic (PLEG): container finished" podID="406afe59-8851-4178-8e7e-fad68db839ad" containerID="441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183" exitCode=0 Nov 03 20:27:33.107541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.106434 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" event={"ID":"406afe59-8851-4178-8e7e-fad68db839ad","Type":"ContainerDied","Data":"441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183"} Nov 03 20:27:33.107541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.106989 5815 scope.go:122] "RemoveContainer" containerID="441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183" Nov 03 20:27:33.125567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.116859 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.125567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.116949 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.125567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.116996 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.158544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.156523 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5"] Nov 03 20:27:33.164205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.164180 5815 generic.go:356] "Generic (PLEG): container finished" podID="7aa8313c-5794-49fd-8e97-caae0f755923" containerID="0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4" exitCode=0 Nov 03 20:27:33.182531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.171730 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq"] Nov 03 20:27:33.198552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.182813 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.198552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.186041 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5"] Nov 03 20:27:33.198552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.186081 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" event={"ID":"7aa8313c-5794-49fd-8e97-caae0f755923","Type":"ContainerDied","Data":"0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4"} Nov 03 20:27:33.198552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.186115 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" event={"ID":"18812c7c-ac28-4334-a508-d73669a0cf0a","Type":"ContainerStarted","Data":"200fc2b65faecb1b93ee6bbec27d833baeb8b020041b024e036c4ff216154549"} Nov 03 20:27:33.198552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.187035 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.198552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.187470 5815 scope.go:122] "RemoveContainer" containerID="0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4" Nov 03 20:27:33.225563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.225531 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" event={"ID":"9fd93595-2633-406e-9d75-5fcf5b1cc835","Type":"ContainerStarted","Data":"e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a"} Nov 03 20:27:33.225832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.225811 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" event={"ID":"9fd93595-2633-406e-9d75-5fcf5b1cc835","Type":"ContainerStarted","Data":"7a3193d60d317a4434e3d1a833c2423d2ff1f8fe05283565e11b0ec1b281bb8e"} Nov 03 20:27:33.257486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.245583 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq"] Nov 03 20:27:33.259881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.258316 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.259881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.258704 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.259881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.258780 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.360973 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361037 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxwr\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361083 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361153 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361193 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361245 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361314 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361489 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361553 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361624 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361652 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361737 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361853 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh597\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361907 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.361959 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.362047 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.362131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.370793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.362171 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.452526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.452170 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podStartSLOduration=2.45214733 podStartE2EDuration="2.45214733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:33.448784447 +0000 UTC m=+1562.979130120" watchObservedRunningTime="2026-11-03 20:27:33.45214733 +0000 UTC m=+1562.982493005" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463492 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463557 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463595 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463619 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463681 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463755 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh597\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463787 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463830 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463894 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463950 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.463982 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464018 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464067 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxwr\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464108 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464161 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464204 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464257 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.466065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.464300 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.468686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.468257 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.469146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.469125 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.473834 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.474094 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.474771 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.474998 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.475126 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.475217 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.475707 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.475839 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.475943 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.476305 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.488542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.481390 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.512536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.510383 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.521593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.521362 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.540731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.528783 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh597\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.672432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.652759 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:33.672432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.653476 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxwr\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.843316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.843272 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:33.851158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:33.851126 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:34.024993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.020850 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:27:34.054537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.046847 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:34.060618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.056748 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:34.060618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.056802 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:34.060618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.059982 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:34.060618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.060030 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:34.372410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.372373 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" event={"ID":"406afe59-8851-4178-8e7e-fad68db839ad","Type":"ContainerStarted","Data":"2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a"} Nov 03 20:27:34.563693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:34.490427 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" event={"ID":"7aa8313c-5794-49fd-8e97-caae0f755923","Type":"ContainerStarted","Data":"8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7"} Nov 03 20:27:35.359539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.353258 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:35.383660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.383622 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.384304 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.383690 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:35.402911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.402797 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.402911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.402849 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:35.406528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.403408 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:27:35.411618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.411590 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.411819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.411797 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:35.424718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.423229 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:35.440141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.431874 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:35.440141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.431966 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:35.440141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.432012 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:35.440141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.437762 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.440141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.437806 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:35.657833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.657802 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:27:35.658176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.658161 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:27:35.699051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.699021 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl"] Nov 03 20:27:35.728187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.728166 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.759724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.743500 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3c25bf\"/\"auth-test-model-3c25bf-kserve-self-signed-certs\"" Nov 03 20:27:35.759724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.743758 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3c25bf\"/\"default-dockercfg-kr96w\"" Nov 03 20:27:35.759724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.743826 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3c25bf\"/\"kube-root-ca.crt\"" Nov 03 20:27:35.759724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.743900 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3c25bf\"/\"openshift-service-ca.crt\"" Nov 03 20:27:35.764540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.763038 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:35.764540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.763132 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:35.764540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.763205 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:27:35.782321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.765275 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.782321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.765320 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:35.823553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.812245 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" event={"ID":"6a461d60-810b-4cdb-9242-c46208bccb7e","Type":"ContainerStarted","Data":"88e2c949c8ce529a84f0e1e603c2a9065a59b04084782e06292af75440a83cd6"} Nov 03 20:27:35.925909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.925854 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:35.927082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.926148 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:27:35.933125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.912810 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.959407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.959068 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.961873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.961438 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.962061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.962042 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zfth\" (UniqueName: \"kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.962576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.962558 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.963065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.963046 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:35.989141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:35.989081 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl"] Nov 03 20:27:36.039531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.034533 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.039531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.034588 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:36.101712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.101423 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.101947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.101929 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zfth\" (UniqueName: \"kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.102635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.102611 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.102796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.102769 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:36.115540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.107849 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.115540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.107899 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.115540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.108106 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.115540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.108250 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.115540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.108788 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.115540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.109039 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.139542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.117449 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.149653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.128447 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.149871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.147788 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.195567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.189015 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq"] Nov 03 20:27:36.225591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.212376 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zfth\" (UniqueName: \"kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth\") pod \"auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.275848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.272255 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.275848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.272313 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:36.321953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.316574 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5"] Nov 03 20:27:36.343625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.343012 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:27:36.358671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.358633 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:36.359435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.359410 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:27:36.383640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.383589 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:27:36.384161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.384140 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:27:36.467668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.423088 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:27:36.467668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.423456 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:27:36.467668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.430581 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:36.467668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.430930 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:36.505610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.502274 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:36.794586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.790628 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.794586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.790689 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:36.933534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.929836 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:27:36.946763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.943614 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:27:36.946763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.943675 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:27:36.946763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.943735 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:27:37.001396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:36.994737 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:37.022565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.022530 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:37.022860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.022811 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:37.193584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.176969 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:37.193584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.177027 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:37.218543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.216986 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:37.218543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.217864 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:27:37.260862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.243211 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:37.386531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.361822 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:37.448595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.390409 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:27:37.448595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.440196 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:37.448595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.448187 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:37.461881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.440240 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:37.490323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.490082 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:37.490632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.490598 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:37.510321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.510274 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" event={"ID":"6a461d60-810b-4cdb-9242-c46208bccb7e","Type":"ContainerStarted","Data":"69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94"} Nov 03 20:27:37.557249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.557213 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" event={"ID":"76d4b78b-0901-41a6-b416-80856daade9c","Type":"ContainerStarted","Data":"778b597b2a3c33ee5a3bd4fccd63e9cb47f6e4c7f2bc137a399d3e41cfeea4d1"} Nov 03 20:27:37.958536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.936372 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:37.961957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:37.961915 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:38.055534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.039938 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.055534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.040019 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:38.055534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.042998 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.055534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.043065 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:38.055534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.044326 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.055534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.044390 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:38.066563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.059339 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.066563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.059390 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:38.066563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.062093 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:27:38.068038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.068018 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:27:38.338736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.230921 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:38.338736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.242598 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:38.396533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.371652 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.397133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.397095 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:38.452836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.452791 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:38.503534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.489275 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:38.503534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.489862 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:38.656441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.656356 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:38.784314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.784265 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.784604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:38.784570 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:39.046159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.046113 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:39.232165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.091550 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.232433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.232398 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:39.232598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.231781 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.232741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.232723 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:39.233309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.091629 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:39.439533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.416126 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:27:39.439533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.416430 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:27:39.439533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.417527 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.439533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.417564 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:39.503759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.503717 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.504065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.504021 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:39.586542 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:27:39.513641 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod636da0b1_49bc_4139_9f5f_111384502bd1.slice/crio-1d0618feff07f9afadd44a3a4505b4bfec221740f3a11e3ac5e413d6ea405051 WatchSource:0}: Error finding container 1d0618feff07f9afadd44a3a4505b4bfec221740f3a11e3ac5e413d6ea405051: Status 404 returned error can't find the container with id 1d0618feff07f9afadd44a3a4505b4bfec221740f3a11e3ac5e413d6ea405051 Nov 03 20:27:39.586542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.544973 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:39.586542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.545012 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:27:39.586542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.545035 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:39.614573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.603111 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:27:39.693554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.686824 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:39.909564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.909103 5815 scope.go:122] "RemoveContainer" containerID="f83951615cb153e7d5aab3b2274b767f2b5ff6e0a2142665ab7cca086c19730d" Nov 03 20:27:39.983563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.960911 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.985214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.985171 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:39.985442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.963237 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.985612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:39.985572 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:40.062686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.062652 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.086703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.086666 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:40.086947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.075854 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" event={"ID":"76d4b78b-0901-41a6-b416-80856daade9c","Type":"ContainerStarted","Data":"42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060"} Nov 03 20:27:40.087076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.063753 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.087265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.087246 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:40.235389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.235286 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:40.236766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.236733 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:27:40.427115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.427072 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:40.444166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.427365 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:40.454724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.442765 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.454724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.454663 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:40.471156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.471113 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:40.575690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.575639 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:40.610534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.598700 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:27:40.610534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.599458 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:27:40.774546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.687752 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:40.774546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.750053 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podStartSLOduration=7.75003219 podStartE2EDuration="7.75003219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:37.674008211 +0000 UTC m=+1567.204353886" watchObservedRunningTime="2026-11-03 20:27:40.75003219 +0000 UTC m=+1570.280377864" Nov 03 20:27:40.820617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.820522 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.820887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.820848 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:40.921538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.907192 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.921538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.907246 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:40.950843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.950807 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.967921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:40.967882 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:41.048530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.041448 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:41.048530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.048500 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.048790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.048578 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:41.052566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.049064 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.052566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.049106 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:41.298137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.298027 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl"] Nov 03 20:27:41.298766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.298690 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:41.301543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.299228 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:27:41.332620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.318749 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.332620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.318802 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:41.458819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.458772 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" event={"ID":"636da0b1-49bc-4139-9f5f-111384502bd1","Type":"ContainerStarted","Data":"1d0618feff07f9afadd44a3a4505b4bfec221740f3a11e3ac5e413d6ea405051"} Nov 03 20:27:41.459474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.459436 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:27:41.473626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.473579 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:41.583723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.583603 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:41.602649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.602606 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:41.603276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.602888 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:41.610531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.606142 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podStartSLOduration=8.60612385 podStartE2EDuration="8.60612385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:40.161456948 +0000 UTC m=+1569.691802623" watchObservedRunningTime="2026-11-03 20:27:41.60612385 +0000 UTC m=+1571.136469534" Nov 03 20:27:41.689194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.689148 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:41.779533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.762613 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:41.779533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.764718 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.779533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.764764 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:41.857328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.857235 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.857618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.857574 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:41.858184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.857839 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.858379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:41.858357 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:42.033298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.033265 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.033630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.033594 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:42.075945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.075909 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.106777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.106735 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:42.260709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.260676 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.261030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.260970 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:42.301794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.301753 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:42.302388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.302036 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:42.461141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.461101 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/health\": context deadline exceeded" Nov 03 20:27:42.479833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.479794 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:27:42.502708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.502661 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:42.564612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.559077 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:27:42.564612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.561709 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:42.604805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.604757 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:42.645646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.645594 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:27:42.646280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.646246 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:27:42.701527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.701457 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:42.835597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.810085 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.836209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.836171 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:42.892531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.862473 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.923656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.923612 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:42.928679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.876075 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.951059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:42.951005 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:43.116592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.084830 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.116592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.088645 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 20:27:43.116592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.108166 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.116592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.108238 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:43.300673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.300631 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.300937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.300699 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:43.323540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.322632 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:27:43.329385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.329348 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:27:43.636530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.623230 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:43.657532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.653082 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:27:43.657532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.653001 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:43.671533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.666804 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:27:43.806352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.805085 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:43.806757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.806725 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.806985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.806939 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:43.920836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.916660 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.920836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.916725 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:43.929010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.921187 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.929010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.921241 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:43.974532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.974469 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.017531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:43.999766 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.058289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.058256 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:44.058593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.058563 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:44.147064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.147033 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:27:44.183589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.180076 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:27:44.257112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.256662 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:44.257112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.256719 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:44.328701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.328658 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.493669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.493620 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.641532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.631883 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.662421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.662379 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.663637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.663605 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:44.668624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.668597 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:27:44.804587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.802686 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:44.804587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.802761 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:44.844920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.825306 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:44.844920 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:27:44.839057 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:27:44.984704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.984659 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:44.985060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:44.984942 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:45.026457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.026412 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:27:45.055577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.055477 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:27:45.078446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.078417 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.078811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.078740 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 20:27:45.083542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.083491 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.083801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.083758 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:45.131991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.131949 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 03 20:27:45.245111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.243821 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:27:45.264999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.264966 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.265313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.265268 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:45.343427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.343317 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:45.526102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.507812 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:27:45.527233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.527211 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:27:45.638728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.638629 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:45.837151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.683301 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:27:45.841529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.683338 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:27:45.856906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.755921 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.872562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.866806 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:45.872562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.836747 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:45.872562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.867946 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.872562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.867984 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:45.872562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.868256 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.872562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:45.868287 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:46.068532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.056465 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:46.256544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.255834 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:46.256544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.255899 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:46.356867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.356760 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:27:46.476661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.476609 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:46.534809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.534770 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:27:46.749990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.749958 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:46.750256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.750228 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:46.838645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.838599 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:46.839578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.839189 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:46.845810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.845782 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:46.845975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.845957 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:46.851909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.851889 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:46.852067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.852049 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:46.859745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.859720 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:46.872017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.871991 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:46.902847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.902823 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:46.904522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.904488 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:46.923735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:46.923712 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:27:47.056745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.056654 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:27:47.314232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.314123 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.374540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.349197 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:47.374540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.349259 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:47.382527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.375126 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.440157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.439839 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.489532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.479604 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.489532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.480187 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:27:47.564749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.564651 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.787532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.784816 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:47.787532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.784872 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:47.878632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.876524 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.879117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.878951 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.888753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.888726 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:47.889021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.888999 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:47.889230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.889215 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:47.889371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.889351 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:47.928764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.927601 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:47.941359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:47.941306 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:48.057951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.057910 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:48.258536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.256781 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body= Nov 03 20:27:48.258536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.256846 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" Nov 03 20:27:48.380130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.380083 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:27:48.482550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.481608 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:48.567315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.567213 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:48.755529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.755107 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 20:27:48.755529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.755205 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 20:27:48.908588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.883745 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:48.908588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.885291 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:48.908588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.885336 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:48.908588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.885732 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:48.908588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.886018 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:48.918370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.914616 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:48.918370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.914677 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:48.931544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.927987 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:48.931544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.930432 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:48.957280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:48.956198 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:49.061726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.061616 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:49.391531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.385412 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:49.441416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.441387 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:27:49.491698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.491645 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:49.551327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.551298 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:49.591022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.582636 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:49.659867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.659782 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:27:49.725830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.725800 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:27:49.852531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.845208 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Nov 03 20:27:49.852531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.845264 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Nov 03 20:27:49.854745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.854718 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:49.854944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.854920 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:49.939582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.915207 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:50.000528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.942269 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:50.000528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:49.942307 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:27:50.224817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.222971 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:27:50.230236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.230214 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:27:50.243866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.243840 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:50.324963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.324926 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:27:50.431099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.430967 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:50.501234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.491878 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:50.672097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.671526 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:27:50.842046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.841954 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:27:50.865531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.856179 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Nov 03 20:27:50.865531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.856251 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Nov 03 20:27:50.982335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:50.982306 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:27:51.006294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.006257 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:51.030821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.030794 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:27:51.077445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.076130 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:51.079477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.079403 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:27:51.288788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.288720 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" event={"ID":"636da0b1-49bc-4139-9f5f-111384502bd1","Type":"ContainerStarted","Data":"676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115"} Nov 03 20:27:51.323181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.323121 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podStartSLOduration=17.32310215 podStartE2EDuration="17.32310215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:51.316795292 +0000 UTC m=+1580.847140964" watchObservedRunningTime="2026-11-03 20:27:51.32310215 +0000 UTC m=+1580.853447824" Nov 03 20:27:51.508644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.508613 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:51.857266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.857239 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:51.858376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.858359 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:27:51.976408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.976344 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5"] Nov 03 20:27:51.982417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.982368 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:51.985124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.985097 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5"] Nov 03 20:27:51.987931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.987892 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"kube-root-ca.crt\"" Nov 03 20:27:51.988323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.988305 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"default-dockercfg-pr7n2\"" Nov 03 20:27:51.988544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.988440 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"openshift-service-ca.crt\"" Nov 03 20:27:51.988667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:51.988640 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"auth-test-model-ba89e0-kserve-self-signed-certs\"" Nov 03 20:27:52.059279 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.059244 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.059457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.059438 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.059617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.059476 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.059617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.059548 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.059702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.059669 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlsbz\" (UniqueName: \"kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.059740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.059711 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.205620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.180817 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.296818 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.296896 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.296942 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.297018 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlsbz\" (UniqueName: \"kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.297070 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.297815 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.306550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.218840 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.327933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.323721 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.439228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.439157 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlsbz\" (UniqueName: \"kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.443161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.443132 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.457068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.452039 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs\") pod \"auth-test-model-ba89e0-kserve-57877fc457-8xnm5\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.598951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.598919 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:52.981484 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:27:52.981432 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod279ab75f_c9e2_4d5a_9907_a608f7633b7a.slice/crio-4fb75b0edcfb7a00be640fa044a9822237ea2002478a9fcefc33ef5191bad0b8 WatchSource:0}: Error finding container 4fb75b0edcfb7a00be640fa044a9822237ea2002478a9fcefc33ef5191bad0b8: Status 404 returned error can't find the container with id 4fb75b0edcfb7a00be640fa044a9822237ea2002478a9fcefc33ef5191bad0b8 Nov 03 20:27:52.981984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:52.981949 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5"] Nov 03 20:27:53.577529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:53.570649 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" event={"ID":"279ab75f-c9e2-4d5a-9907-a608f7633b7a","Type":"ContainerStarted","Data":"58f9322fc4903e3e0844a1a7796430f1607e64b7128dd2663aefde73fea22663"} Nov 03 20:27:53.577529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:53.570695 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" event={"ID":"279ab75f-c9e2-4d5a-9907-a608f7633b7a","Type":"ContainerStarted","Data":"4fb75b0edcfb7a00be640fa044a9822237ea2002478a9fcefc33ef5191bad0b8"} Nov 03 20:27:53.595181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:53.595127 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podStartSLOduration=2.595113487 podStartE2EDuration="2.595113487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:53.592706192 +0000 UTC m=+1583.123051864" watchObservedRunningTime="2026-11-03 20:27:53.595113487 +0000 UTC m=+1583.125459178" Nov 03 20:27:53.599576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:53.599547 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:27:56.510528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:56.508912 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:57.010867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.010835 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:57.095661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.095611 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:57.235757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.235702 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:57.450602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.439166 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:57.450602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.442588 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:57.494534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.493791 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:57.513738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.510860 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:57.526532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:57.522929 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:58.057177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.057118 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:58.098900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.098859 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:58.099722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.099699 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:27:58.260670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.260623 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:27:58.261440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.261416 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:27:58.505968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.494923 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:58.505968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.495648 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:27:58.653700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.653007 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:58.854592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.838072 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:27:58.881552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:58.880659 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:27:59.058988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.058924 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:59.099589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.099547 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:27:59.490658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.490609 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:59.503266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.502588 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:59.504307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.504277 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:59.608238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.607643 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:59.657138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.657093 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:27:59.838712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.837957 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:27:59.896533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:27:59.888590 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:00.127582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.116283 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:00.449247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.425049 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:00.513564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.509818 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:00.513564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.510536 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:28:00.562796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.562738 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:00.625693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.572241 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:00.630772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.630735 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:28:00.631288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.631262 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:28:00.665609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.665555 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:00.905811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:00.905757 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:01.001529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.000635 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:01.001529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.000695 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:01.001529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.000733 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:01.060543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.059741 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:28:01.089531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.064704 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:01.176292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.165178 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:01.358532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.334007 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:01.358532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.335671 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:28:01.710528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.710004 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:28:01.999584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:01.999195 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:02.008500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.005602 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:02.189742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.188590 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:02.339060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.338954 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:02.449611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.449568 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:02.511529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.510871 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:02.627338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.627258 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:28:02.686697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.686653 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:02.715535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.711759 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:02.976907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:02.976880 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:28:03.380860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.379724 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:28:03.511213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.511160 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:03.511828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.511792 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:28:03.693720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.693621 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:03.694530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.694486 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:03.808978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.808938 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:03.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:03.809765 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:28:04.135532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.123784 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:04.136532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.136159 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.169229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.168641 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.186562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.180136 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.345569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.339597 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.386951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.384965 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.386951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.385814 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:04.543532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.530050 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.707586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.696716 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:04.723655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.718791 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:04.827659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:04.827611 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:05.129271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.129222 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:05.132123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.130923 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:05.172497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.172460 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:05.175147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.175124 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:28:05.232529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.230742 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:05.232529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.231288 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:05.348645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.347757 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:28:05.349938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.349907 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:28:05.544741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.541641 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:05.729847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.702362 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:05.900099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.900048 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:05.906200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.906168 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:28:05.982064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:05.982036 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:28:06.075678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:06.075651 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:06.076372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:06.076351 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:28:06.085246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:06.085225 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:06.088414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:06.088384 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:08.250743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:08.250683 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:28:08.251293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:08.250789 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:08.753962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:08.753643 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:28:08.753962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:08.753714 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:10.786137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.760233 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw"] Nov 03 20:28:10.816073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.816028 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw"] Nov 03 20:28:10.816439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.816415 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:10.828601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.828572 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"openshift-service-ca.crt\"" Nov 03 20:28:10.828742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.828690 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"auth-test-model-bab2c3-kserve-self-signed-certs\"" Nov 03 20:28:10.828811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.828780 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"default-dockercfg-vvf7p\"" Nov 03 20:28:10.828870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.828845 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"kube-root-ca.crt\"" Nov 03 20:28:10.872806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.872771 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdpgw\" (UniqueName: \"kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.872913 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.873098 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.873143 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.873219 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.873273 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975068 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975132 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975205 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975261 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975330 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdpgw\" (UniqueName: \"kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975453 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.975958 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.988338 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:10.989725 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.019235 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.048813 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.106371 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdpgw\" (UniqueName: \"kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw\") pod \"auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.186286 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:28:11.570422 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod744520ef_cd92_40a9_a060_c97a0af7c33e.slice/crio-7763715c7f1d7adee965c88761cd7d6dae77a979c274041717e750f9f86e5af6 WatchSource:0}: Error finding container 7763715c7f1d7adee965c88761cd7d6dae77a979c274041717e750f9f86e5af6: Status 404 returned error can't find the container with id 7763715c7f1d7adee965c88761cd7d6dae77a979c274041717e750f9f86e5af6 Nov 03 20:28:11.746261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.571857 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw"] Nov 03 20:28:11.867040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.851714 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" event={"ID":"744520ef-cd92-40a9-a060-c97a0af7c33e","Type":"ContainerStarted","Data":"9bfcfef7f975db1f7b6ffbf555054fdeab4f883ab25e938202c53c075f9b615e"} Nov 03 20:28:11.867040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:11.851765 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" event={"ID":"744520ef-cd92-40a9-a060-c97a0af7c33e","Type":"ContainerStarted","Data":"7763715c7f1d7adee965c88761cd7d6dae77a979c274041717e750f9f86e5af6"} Nov 03 20:28:12.321042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.187724 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:12.863410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.863327 5815 generic.go:356] "Generic (PLEG): container finished" podID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerID="97d792a1ff7646d96b29370c2caf519086eee8a8794b18c64b119a79262dcc73" exitCode=0 Nov 03 20:28:12.863569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.863543 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" event={"ID":"b5ba818e-adb1-4432-ac0a-ca912d3c28c1","Type":"ContainerDied","Data":"97d792a1ff7646d96b29370c2caf519086eee8a8794b18c64b119a79262dcc73"} Nov 03 20:28:12.863905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.863856 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": read tcp 10.132.0.2:36442->10.132.0.199:15021: read: connection reset by peer" start-of-body= Nov 03 20:28:12.864003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.863909 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": read tcp 10.132.0.2:36442->10.132.0.199:15021: read: connection reset by peer" Nov 03 20:28:12.864103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.864083 5815 scope.go:122] "RemoveContainer" containerID="97d792a1ff7646d96b29370c2caf519086eee8a8794b18c64b119a79262dcc73" Nov 03 20:28:12.882989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.882945 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:12.883322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.883047 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:12.883322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.883089 5815 generic.go:356] "Generic (PLEG): container finished" podID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerID="a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246" exitCode=0 Nov 03 20:28:12.883322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.883097 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:12.883322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.883161 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" event={"ID":"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0","Type":"ContainerDied","Data":"a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246"} Nov 03 20:28:12.883782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.883760 5815 scope.go:122] "RemoveContainer" containerID="a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246" Nov 03 20:28:12.885238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.885207 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:12.885362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.885270 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:12.885362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.885302 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:12.947761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:12.947717 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podStartSLOduration=2.9477032640000003 podStartE2EDuration="2.947703264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:11.872636941 +0000 UTC m=+1601.402982612" watchObservedRunningTime="2026-11-03 20:28:12.947703264 +0000 UTC m=+1602.478048976" Nov 03 20:28:13.858023 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:28:13.857982 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:28:13.927013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.926966 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" event={"ID":"b5ba818e-adb1-4432-ac0a-ca912d3c28c1","Type":"ContainerStarted","Data":"be350a5510355906c146c510cde1a6d430bf500e4e0996d6319ffbcf04768dac"} Nov 03 20:28:13.932532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.932383 5815 generic.go:356] "Generic (PLEG): container finished" podID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerID="838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380" exitCode=0 Nov 03 20:28:13.932532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.932498 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" event={"ID":"65d4bf87-dd67-4905-bf8f-bb093e231c4d","Type":"ContainerDied","Data":"838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380"} Nov 03 20:28:13.932932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.932914 5815 scope.go:122] "RemoveContainer" containerID="838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380" Nov 03 20:28:13.935864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.935837 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:13.936106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.936089 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:13.936369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.936352 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:13.936840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.936824 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" event={"ID":"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0","Type":"ContainerStarted","Data":"65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215"} Nov 03 20:28:13.949377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:13.949358 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:28:14.754529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.751085 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4"] Nov 03 20:28:14.756835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.756211 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:28:14.756835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.756326 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:14.761812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.759935 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:14.761812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.760008 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:14.791534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.778352 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-3211a0\"/\"default-dockercfg-8mjms\"" Nov 03 20:28:14.791534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.778477 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-3211a0\"/\"praxis-route-3211a0-kserve-self-signed-certs\"" Nov 03 20:28:14.791534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.778787 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-3211a0\"/\"kube-root-ca.crt\"" Nov 03 20:28:14.791534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:14.778849 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-3211a0\"/\"openshift-service-ca.crt\"" Nov 03 20:28:15.077584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.072047 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:15.077584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.072112 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.105084 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4"] Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.108375 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.144347 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxj9d\" (UniqueName: \"kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.144485 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.144557 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.144718 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.177850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.173098 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.324384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.304828 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxj9d\" (UniqueName: \"kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.324834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.324808 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.325011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.324995 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.325229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.325215 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.325587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.325567 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.325799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.325783 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.349543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.345992 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.349543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.346268 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.349543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.346564 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.349543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.347844 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.405868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.402952 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.455925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.451687 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxj9d\" (UniqueName: \"kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d\") pod \"praxis-route-3211a0-kserve-785759cc4f-r27c4\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.561532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.555772 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" event={"ID":"65d4bf87-dd67-4905-bf8f-bb093e231c4d","Type":"ContainerStarted","Data":"8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2"} Nov 03 20:28:15.578623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.577873 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:15.759536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.759275 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:15.759536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.759331 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:15.954590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.954198 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:15.954590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:15.954268 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:16.064816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:16.056453 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:28:16.064816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:16.056583 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:16.064816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:16.056626 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:17.088535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.067622 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:17.088535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.067679 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:17.119533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.089584 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:17.169579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.159686 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:17.169579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.159748 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:17.169579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.160182 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:17.169579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.160225 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:17.193578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.193546 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4"] Nov 03 20:28:17.287533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.269723 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225"] Nov 03 20:28:17.327342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.327310 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225"] Nov 03 20:28:17.341622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.340266 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.492768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.492726 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.493029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.493010 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.493208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.493189 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.493385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.493369 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.493698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.493675 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.493847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.493830 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.493993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.493979 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.494154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.494141 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.494411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.494393 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgkg\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.633540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.621583 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.686828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.686700 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.702008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.629288 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.702326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.688117 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.702646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.702627 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.703457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.703426 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.704024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.704004 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgkg\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.704025 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.716735 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.716882 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.717030 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.717449 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.717716 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.721558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.719738 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.749198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.749127 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.749592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.749418 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:17.750002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.749452 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:17.763885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.763857 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.801748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.801713 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:17.802068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.802020 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:17.817320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.817282 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgkg\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.837067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.837034 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:17.966530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.965726 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:17.966530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.965803 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:17.995735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.995673 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": context deadline exceeded" Nov 03 20:28:17.996256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:17.996221 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:18.000937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.000861 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:18.053887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.053855 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:18.054133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.054107 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:18.065730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.065687 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:18.098545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.096973 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:28:18.098545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.097330 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:28:18.098545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.097683 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:28:18.395631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.391625 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:18.732555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.728137 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:18.753406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.752946 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:18.753406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.753003 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:18.926048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.925093 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:28:18.963711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.963630 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:18.964045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.964014 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:18.966121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:18.966097 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87"] Nov 03 20:28:19.003011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.000011 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": context deadline exceeded" Nov 03 20:28:19.023142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.003969 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.023142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.003977 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:28:19.025915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.025851 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" event={"ID":"d88aa3a5-5e38-4cba-9cda-80134f0398a3","Type":"ContainerStarted","Data":"be82cba0b1c661c8118ca665b0f2c45b8c3ab124ce982e2427b5099a758ed0ad"} Nov 03 20:28:19.070539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.065491 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:19.070539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.065584 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:19.070539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.066784 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:19.070539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.067135 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:28:19.102726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.102159 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:28:19.102726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.102590 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:28:19.103290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.102892 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:19.121577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.110552 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft"] Nov 03 20:28:19.392608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.392467 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:19.426874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.426836 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:19.427073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.426891 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87"] Nov 03 20:28:19.440584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.427356 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460136 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ksj\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460247 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460355 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460401 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460438 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460622 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460710 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460893 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.461568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.460972 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.604543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.604492 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646310 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646383 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646425 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646476 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646576 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646617 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646709 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646746 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646790 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646902 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.646960 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647082 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647144 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647191 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ksj\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647227 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647284 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647315 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4sh5\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647364 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.647998 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.653616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.648272 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.693157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.692348 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.694141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.694103 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.694759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.694725 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.835759 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.870574 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ksj\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.870869 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.871667 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.871861 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.878986 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.879275 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.883375 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:19.902875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.883881 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:19.948592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:19.931095 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft"] Nov 03 20:28:20.137669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.010738 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:20.160938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.043500 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.161325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.161299 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.161631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.161557 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.161840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.161821 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.162114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.162087 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.162449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.162424 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.162639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.162623 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.162790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.162774 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4sh5\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.163100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.163080 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.164026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.164003 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.164440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.053013 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.164663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.164631 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:20.186559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.184362 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:28:20.203622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.199575 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:20.314536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.290759 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.331956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.331925 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.335103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.335078 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.391005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.291499 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.435541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.435449 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:20.513542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.500243 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:20.573558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.518680 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.573558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.518758 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:20.631204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.630557 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:20.647331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.647293 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4sh5\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.656098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.653926 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.683820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.674210 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:20.683820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.675469 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.737523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.737404 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.762552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.761600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:20.771592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.763607 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.771592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.763672 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 20:28:20.792534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.788891 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:20.885501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.885448 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:20.889612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.889584 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:28:20.937869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.937608 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:20.960845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.960815 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:28:20.961187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.961165 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" event={"ID":"d88aa3a5-5e38-4cba-9cda-80134f0398a3","Type":"ContainerStarted","Data":"639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4"} Nov 03 20:28:20.961381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.961365 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:20.961563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.961551 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:20.983874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.983827 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:28:20.987979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.987942 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.989916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.989662 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 03 20:28:20.998685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:20.990455 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:21.058637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.056267 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:21.058637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.056729 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.058637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.056775 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:21.081567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.080362 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.091288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.091254 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:21.096530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.096161 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:28:21.142733 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:28:21.142683 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb44027_53ca_4bd8_8d11_fb1909ecf87d.slice/crio-ff03d40c995fa840e3f5ca8ca71c0d60bb1b82c0487565c04dca084bafcbf33d WatchSource:0}: Error finding container ff03d40c995fa840e3f5ca8ca71c0d60bb1b82c0487565c04dca084bafcbf33d: Status 404 returned error can't find the container with id ff03d40c995fa840e3f5ca8ca71c0d60bb1b82c0487565c04dca084bafcbf33d Nov 03 20:28:21.178573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.158078 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:21.178573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.158176 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:21.178573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.158221 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:21.235344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.220215 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:21.309652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.309562 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podStartSLOduration=7.309544452 podStartE2EDuration="7.309544452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:21.102966959 +0000 UTC m=+1610.633312631" watchObservedRunningTime="2026-11-03 20:28:21.309544452 +0000 UTC m=+1610.839890126" Nov 03 20:28:21.334145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.334037 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225"] Nov 03 20:28:21.345989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.339985 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:21.591930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.591903 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:21.769552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.766863 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:28:21.769552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.768571 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:28:21.770017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.769972 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 20:28:21.963988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.963952 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:28:21.965286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:21.965264 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:28:22.025242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.024487 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:28:22.031534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.031464 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:22.031741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.031719 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:22.031891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.031872 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:22.052010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.051662 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.052674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.052182 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:22.118946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.110950 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" event={"ID":"1cb44027-53ca-4bd8-8d11-fb1909ecf87d","Type":"ContainerStarted","Data":"d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699"} Nov 03 20:28:22.118946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.111071 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" event={"ID":"1cb44027-53ca-4bd8-8d11-fb1909ecf87d","Type":"ContainerStarted","Data":"ff03d40c995fa840e3f5ca8ca71c0d60bb1b82c0487565c04dca084bafcbf33d"} Nov 03 20:28:22.134794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.134763 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87"] Nov 03 20:28:22.140891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.139966 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:22.140891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.140111 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:22.140891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.140205 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:22.259627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.252014 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:22.321230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.321207 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft"] Nov 03 20:28:22.343179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:22.342731 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podStartSLOduration=6.342710038 podStartE2EDuration="6.342710038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:22.260729015 +0000 UTC m=+1611.791074695" watchObservedRunningTime="2026-11-03 20:28:22.342710038 +0000 UTC m=+1611.873055718" Nov 03 20:28:23.099536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.058617 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:23.099536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.058738 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.099536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.058790 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:23.099536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.059977 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.099536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.060021 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Nov 03 20:28:23.369431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.369346 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" event={"ID":"d721d008-e6c7-4c0c-86c5-f28e73bb6661","Type":"ContainerStarted","Data":"fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043"} Nov 03 20:28:23.372746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.372717 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" event={"ID":"d721d008-e6c7-4c0c-86c5-f28e73bb6661","Type":"ContainerStarted","Data":"d92a9ae85f9832365ae4f676298a54d7b9da2d8365b90c09befdcdd4674f64c6"} Nov 03 20:28:23.389682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.389645 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" event={"ID":"b19cd843-43af-48c3-b4a8-8d268bfd5f35","Type":"ContainerStarted","Data":"e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8"} Nov 03 20:28:23.389974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.389952 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" event={"ID":"b19cd843-43af-48c3-b4a8-8d268bfd5f35","Type":"ContainerStarted","Data":"b6cf222b8eb9691a6930b5870af92b1fe9a1862ab11775c504eaea9c3a4cf1da"} Nov 03 20:28:23.621708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.621567 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podStartSLOduration=5.621547506 podStartE2EDuration="5.621547506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:23.426180565 +0000 UTC m=+1612.956526238" watchObservedRunningTime="2026-11-03 20:28:23.621547506 +0000 UTC m=+1613.151893179" Nov 03 20:28:23.683219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.683155 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podStartSLOduration=5.683132441 podStartE2EDuration="5.683132441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:23.455570982 +0000 UTC m=+1612.985916654" watchObservedRunningTime="2026-11-03 20:28:23.683132441 +0000 UTC m=+1613.213478113" Nov 03 20:28:23.867702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.866748 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:23.900596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.891981 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.900596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.892042 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:23.970555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:23.965768 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:24.027538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.020296 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:24.027538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.020355 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:24.045077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.041587 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:24.045077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.041648 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:24.097532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.085203 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:28:24.116877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.116838 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:28:24.148553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.148342 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 20:28:24.768528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.763627 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:24.785528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.781269 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:24.869483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.869438 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:24.891593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.886242 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:24.907862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.907808 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:24.944853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.944817 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:24.945212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.945179 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:24.987540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.987460 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:24.998691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:24.998653 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.040711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.040617 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.040948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.040922 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:25.236530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.229732 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.583830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.583789 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:25.765649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.765600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.766485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.766458 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:25.781621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.781585 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.782493 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.782470 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.921823 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.921885 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.926809 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.927914 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.928215 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.941210 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.943587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.941248 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:25.995110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:25.995069 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.008308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.007754 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.008943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.008922 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:26.029934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.029884 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:28:26.069648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.066174 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:26.069648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.066752 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.069648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.066797 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:26.101603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.095902 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.139533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.138582 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": context deadline exceeded" Nov 03 20:28:26.284739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.274420 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:26.313575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.306607 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.785662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.782118 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.871656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.871620 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.872019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.871990 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:26.895908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.895870 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:28:26.945535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.939634 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.974553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.972301 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.974553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:26.972360 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:27.007051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.007019 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:27.038608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.038495 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:27.063940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.034233 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:28:27.069528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.034027 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": context deadline exceeded" Nov 03 20:28:27.079650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.079612 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.151534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.133081 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.151534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.133607 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:27.167531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.159718 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.167531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.160420 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:27.297772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.268211 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.298128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.298087 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.485715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.485622 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.511660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.511615 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.632600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.628797 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc"] Nov 03 20:28:27.641048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.641022 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:27.650532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.645588 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.672336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.647773 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.678630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.675273 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-30e9de\"/\"openshift-service-ca.crt\"" Nov 03 20:28:27.678630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.676223 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-30e9de\"/\"auth-test-model-30e9de-kserve-self-signed-certs\"" Nov 03 20:28:27.678630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.676352 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-30e9de\"/\"default-dockercfg-jz88g\"" Nov 03 20:28:27.678630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.676419 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-30e9de\"/\"kube-root-ca.crt\"" Nov 03 20:28:27.699534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.696550 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvwf\" (UniqueName: \"kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.704533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.696871 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.704533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.701393 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.704533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.701627 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.726245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.718345 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.726245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.718495 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.798908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.787140 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:27.819524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.819463 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.851806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.851770 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.866261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.852969 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:27.866261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.861655 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:27.866261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.865901 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:27.912168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.866782 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.912168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.905080 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.929731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.929685 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.930627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.930034 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:28:27.931790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.931766 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.932029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.932012 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.932236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.932220 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvwf\" (UniqueName: \"kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.932412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.932399 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:27.932597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.932583 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:28.160579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.933216 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:28.160579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:27.933432 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:28.160579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.078404 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:28.160579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.079939 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:28.160579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.111209 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:28.160579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.111255 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc"] Nov 03 20:28:28.188588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.179905 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:28.188588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.179999 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:28.188588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.185224 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:28.188588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.185291 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:28.188588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.185936 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:28.206765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.206710 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:28:28.232626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.207534 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:28:28.571190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.571161 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:28.743533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.620240 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:28.743533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.741947 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:28.834588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.802976 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:28.847293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.847251 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 03 20:28:28.917479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.914647 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:28.969469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.968851 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:28.969469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.968926 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:29.004557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.973955 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:28:29.004557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.978993 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:29.004557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:28.979046 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:29.027777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.005721 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvwf\" (UniqueName: \"kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf\") pod \"auth-test-model-30e9de-kserve-5dd47c8965-slwkc\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:29.027777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.011663 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:29.027777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.011734 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:29.101769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.101674 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.128163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.128117 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.129936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.129909 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:29.232775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.223234 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.232775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.223587 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:28:29.338770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.338721 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.533531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.508144 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:29.533921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.533891 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:28:29.534142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.534128 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:29.665999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.665960 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:29.834588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.815695 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.834588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.816145 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.869048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.868261 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": context deadline exceeded" Nov 03 20:28:29.932263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.932220 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:29.958906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.958870 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:29.959189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.959156 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:29.959409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.959394 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:30.002531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:29.991818 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:30.113583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.108656 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:30.113583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.109329 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:30.143116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.143083 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:30.143390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.143361 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:30.143619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.143594 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:30.145191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.145170 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:30.145350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.145331 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:30.225214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.225171 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:30.226555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.226526 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:28:30.688605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.678595 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:28:30.688605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.679134 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Nov 03 20:28:30.751395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.731717 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:28:30.826692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.826641 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:30.826947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.826914 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:30.874632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.870662 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:30.946593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:30.943634 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": context deadline exceeded" Nov 03 20:28:31.031123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.031083 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:31.114533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.110661 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:31.193795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.187460 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:31.193795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.187533 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:31.232730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.232662 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:31.261707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.261660 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:31.319890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.319843 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:28:31.363619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.363578 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:31.379672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.376240 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Nov 03 20:28:31.379672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.379215 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Nov 03 20:28:31.379672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.379008 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Nov 03 20:28:31.379672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.379556 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Nov 03 20:28:31.390215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.390180 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" event={"ID":"70d684d2-cade-4c87-a6c7-2f138c02272b","Type":"ContainerStarted","Data":"70b26ef8d01815e1939f1f340616ce3a04e6f587886460530d5a3c74b703f1b1"} Nov 03 20:28:31.463632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.463534 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:28:31.559579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.552946 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:28:31.613538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.610138 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:28:31.997078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.997039 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Nov 03 20:28:31.997713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:31.997681 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Nov 03 20:28:32.164743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.164714 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:28:32.166958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.166939 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:28:32.187414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.181799 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:32.187414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.182751 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded" Nov 03 20:28:32.187414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.183010 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:32.466288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.380691 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:28:32.472688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.472642 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": context deadline exceeded" Nov 03 20:28:32.528473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.501391 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:32.529386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.501640 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:32.557077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.557045 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:32.559246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.559227 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:32.574722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.574694 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:28:32.578573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.578544 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:28:32.607548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.606908 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:32.616310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.610358 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:32.638332 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:28:32.637056 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod636da0b1_49bc_4139_9f5f_111384502bd1.slice/crio-1d0618feff07f9afadd44a3a4505b4bfec221740f3a11e3ac5e413d6ea405051\": RecentStats: unable to find data in memory cache]" Nov 03 20:28:32.670725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.663231 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:28:32.670725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.663397 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:32.670725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.665578 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:32.718973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.699109 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc"] Nov 03 20:28:32.718973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.701098 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:28:32.718973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.703065 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:28:32.718973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.707635 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:28:32.718973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.708122 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:28:32.728880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.728847 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8"] Nov 03 20:28:32.736771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.736749 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:28:32.737538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.737197 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.743263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.742177 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8"] Nov 03 20:28:32.744265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.743697 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"kube-root-ca.crt\"" Nov 03 20:28:32.744265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.743795 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"openshift-service-ca.crt\"" Nov 03 20:28:32.744265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.743887 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"default-dockercfg-tzxvn\"" Nov 03 20:28:32.745669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.745326 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"auth-test-model-99a8fe-kserve-self-signed-certs\"" Nov 03 20:28:32.844255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.844179 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.844682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.844654 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.844929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.844908 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hks7x\" (UniqueName: \"kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.845548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.845486 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.845811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.845789 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.845903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.845883 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.916956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.916606 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:32.917963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.917894 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:32.920070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.920050 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:28:32.960134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960098 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.960337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960164 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hks7x\" (UniqueName: \"kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.960337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960194 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.960566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960414 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.960566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960501 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.960696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960500 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.960833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.960714 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.961134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.961111 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.961202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.961148 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:32.969437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:32.969406 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:33.019578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.017715 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:33.020700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.020672 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hks7x\" (UniqueName: \"kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x\") pod \"auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:33.066409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.066381 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:33.224085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.224047 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7"] Nov 03 20:28:33.260535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.251729 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7"] Nov 03 20:28:33.260535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.251872 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351319 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351380 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351492 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351569 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351765 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351795 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351961 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.351989 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gswmv\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.372480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.352020 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.453105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.453074 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.454833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.454807 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.455373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.455353 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.455879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.455860 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.456132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.456111 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.456722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.456701 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswmv\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.456929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.453728 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.456929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.456862 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.457436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.455242 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.457436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.456647 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.457944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.457903 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.458085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.458066 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.458263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.458248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.458344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.458335 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.458665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.458649 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.461211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.461184 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.472970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.472635 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.482915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.482888 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswmv\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.632305 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:28:33.632275 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc916591_d0b9_48e3_a940_e605f410ccb5.slice/crio-2a54f43f1097e0f66639c42b4e747e7b50d70dec1643b36d2ac52dcb749d0bf9 WatchSource:0}: Error finding container 2a54f43f1097e0f66639c42b4e747e7b50d70dec1643b36d2ac52dcb749d0bf9: Status 404 returned error can't find the container with id 2a54f43f1097e0f66639c42b4e747e7b50d70dec1643b36d2ac52dcb749d0bf9 Nov 03 20:28:33.632981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.632959 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8"] Nov 03 20:28:33.699059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.699023 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" event={"ID":"70d684d2-cade-4c87-a6c7-2f138c02272b","Type":"ContainerStarted","Data":"3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8"} Nov 03 20:28:33.701055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.700906 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" event={"ID":"bc916591-d0b9-48e3-a940-e605f410ccb5","Type":"ContainerStarted","Data":"2a54f43f1097e0f66639c42b4e747e7b50d70dec1643b36d2ac52dcb749d0bf9"} Nov 03 20:28:33.759583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.746178 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:33.818554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:33.815708 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" podStartSLOduration=7.815686176 podStartE2EDuration="7.815686176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:33.745898849 +0000 UTC m=+1623.276244551" watchObservedRunningTime="2026-11-03 20:28:33.815686176 +0000 UTC m=+1623.346031849" Nov 03 20:28:34.138131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.138103 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:34.314529 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:28:34.314485 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbaa43a_6433_408b_9c35_87a7a44a6169.slice/crio-f814a2457fec7f04f18bce8fc2284f8b85f4dfb8a9950160372f47a542b10488 WatchSource:0}: Error finding container f814a2457fec7f04f18bce8fc2284f8b85f4dfb8a9950160372f47a542b10488: Status 404 returned error can't find the container with id f814a2457fec7f04f18bce8fc2284f8b85f4dfb8a9950160372f47a542b10488 Nov 03 20:28:34.318161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.318123 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:34.318272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.318204 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:34.318272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.318249 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:28:34.320797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.320772 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7"] Nov 03 20:28:34.721129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.721091 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" event={"ID":"abbaa43a-6433-408b-9c35-87a7a44a6169","Type":"ContainerStarted","Data":"f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d"} Nov 03 20:28:34.721309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.721157 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" event={"ID":"abbaa43a-6433-408b-9c35-87a7a44a6169","Type":"ContainerStarted","Data":"f814a2457fec7f04f18bce8fc2284f8b85f4dfb8a9950160372f47a542b10488"} Nov 03 20:28:34.723404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.723378 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" event={"ID":"bc916591-d0b9-48e3-a940-e605f410ccb5","Type":"ContainerStarted","Data":"1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9"} Nov 03 20:28:34.746786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.746761 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:34.747156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.747119 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podStartSLOduration=1.747107382 podStartE2EDuration="1.747107382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:34.743366664 +0000 UTC m=+1624.273712336" watchObservedRunningTime="2026-11-03 20:28:34.747107382 +0000 UTC m=+1624.277453054" Nov 03 20:28:34.777294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:34.777242 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podStartSLOduration=3.777223744 podStartE2EDuration="3.777223744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:34.769320966 +0000 UTC m=+1624.299666638" watchObservedRunningTime="2026-11-03 20:28:34.777223744 +0000 UTC m=+1624.307569421" Nov 03 20:28:35.070623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:35.070494 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:35.748599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:35.748570 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:35.749074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:35.748624 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:36.747271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:36.747245 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:36.747446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:36.747289 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:37.747813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:37.747781 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:37.748198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:37.747831 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:38.747498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:38.747469 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:38.747689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:38.747525 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:38.858578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:38.855169 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:28:39.158591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:39.154846 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:39.335173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:39.335146 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:39.747596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:39.747562 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:39.747763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:39.747611 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:39.948935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:39.947129 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:28:40.848776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:40.848747 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:40.849092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:40.849059 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:41.545533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:41.513208 5815 scope.go:122] "RemoveContainer" containerID="4e2660ff0b2d246b1249cd779ffc3743046609a8f7a436433a448fcfdf57cd8b" Nov 03 20:28:41.750769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:41.750725 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 03 20:28:41.751106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:41.751073 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 03 20:28:41.955745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:41.955701 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:41.983628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:41.983584 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:42.118726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:42.118666 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:42.251533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:42.242672 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:42.251533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:42.243064 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:28:42.419536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:42.400866 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:43.074532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:43.073713 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:43.097255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:43.097217 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:43.324897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:43.324818 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:43.325418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:43.325388 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:43.327133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:43.327106 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:28:43.657934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:43.657856 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:28:43.859305 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:28:43.859263 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:28:48.750910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.750872 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j"] Nov 03 20:28:48.760275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.760252 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:48.767784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.766153 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j"] Nov 03 20:28:48.767784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.767523 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"openshift-service-ca.crt\"" Nov 03 20:28:48.767784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.767615 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"default-dockercfg-jhptg\"" Nov 03 20:28:48.767784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.767687 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"auth-test-model-6e5139-kserve-self-signed-certs\"" Nov 03 20:28:48.767784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.767751 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"kube-root-ca.crt\"" Nov 03 20:28:48.897720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.897663 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:48.897907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.897741 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:48.897907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.897799 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:48.898120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.898077 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:48.898176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.898141 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxkp\" (UniqueName: \"kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:48.898262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:48.898249 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.021257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.006732 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.021257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.006800 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.021257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.006846 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.021257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.007093 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.021257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.007122 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxkp\" (UniqueName: \"kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.021257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.007227 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.022240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.022208 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.022688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.022667 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.024676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.023076 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.078534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.067805 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.094328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.094248 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.114076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.101135 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxkp\" (UniqueName: \"kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp\") pod \"auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.383451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.383366 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:49.747006 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:28:49.746975 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1449fe1f_bbae_48c3_a531_4dea8c03c441.slice/crio-6c4793d60fcab367f428b9cfe61d77395202be5d6f5f083fe15ec3f8b6195acc WatchSource:0}: Error finding container 6c4793d60fcab367f428b9cfe61d77395202be5d6f5f083fe15ec3f8b6195acc: Status 404 returned error can't find the container with id 6c4793d60fcab367f428b9cfe61d77395202be5d6f5f083fe15ec3f8b6195acc Nov 03 20:28:49.747987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.747964 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j"] Nov 03 20:28:49.788676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:49.788646 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" event={"ID":"1449fe1f-bbae-48c3-a531-4dea8c03c441","Type":"ContainerStarted","Data":"6c4793d60fcab367f428b9cfe61d77395202be5d6f5f083fe15ec3f8b6195acc"} Nov 03 20:28:50.802729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:50.802689 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" event={"ID":"1449fe1f-bbae-48c3-a531-4dea8c03c441","Type":"ContainerStarted","Data":"407ad0341f37759e98b15adff63d2a89d531a88dcf1edceb0c52d653ac54dc22"} Nov 03 20:28:50.828608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:50.828559 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" podStartSLOduration=2.828541362 podStartE2EDuration="2.828541362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:50.822867812 +0000 UTC m=+1640.353213481" watchObservedRunningTime="2026-11-03 20:28:50.828541362 +0000 UTC m=+1640.358887037" Nov 03 20:28:51.385211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:51.385176 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:59.387158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:59.387127 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:28:59.401338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:28:59.401310 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:29:00.010344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:00.010312 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:29:00.540856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:00.540827 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv"] Nov 03 20:29:00.541359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:00.541110 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" containerID="cri-o://3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251" gracePeriod=30 Nov 03 20:29:05.952877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:05.952849 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:29:06.057357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057326 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057380 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057453 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057578 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057740 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057792 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057866 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.057939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057872 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:06.057939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.057925 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.058103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.058002 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo\") pod \"67b7dce9-2229-44e9-b85f-279a9092dece\" (UID: \"67b7dce9-2229-44e9-b85f-279a9092dece\") " Nov 03 20:29:06.058343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.058314 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:06.058446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.058389 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:06.058988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.058831 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:06.058988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.058917 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:06.059341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.059251 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.059341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.059278 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.059341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.059292 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.059341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.059307 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.059341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.059320 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b7dce9-2229-44e9-b85f-279a9092dece-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.060439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.060407 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:06.061078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.061055 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:06.062095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.062070 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:06.062229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.062195 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt" pod "67b7dce9-2229-44e9-b85f-279a9092dece" (UID: "67b7dce9-2229-44e9-b85f-279a9092dece"). InnerVolumeSpecName "kube-api-access-mmvvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:06.150672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.150638 5815 generic.go:356] "Generic (PLEG): container finished" podID="67b7dce9-2229-44e9-b85f-279a9092dece" containerID="3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251" exitCode=0 Nov 03 20:29:06.150841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.150760 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" event={"ID":"67b7dce9-2229-44e9-b85f-279a9092dece","Type":"ContainerDied","Data":"3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251"} Nov 03 20:29:06.150841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.150798 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" event={"ID":"67b7dce9-2229-44e9-b85f-279a9092dece","Type":"ContainerDied","Data":"25c6afe17eaf55711b0f7ab66493e5ebbce57f86cf142c1b35bc67f16975fa99"} Nov 03 20:29:06.150841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.150817 5815 scope.go:122] "RemoveContainer" containerID="3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251" Nov 03 20:29:06.150952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.150940 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv" Nov 03 20:29:06.161327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.161297 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.161542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.161331 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b7dce9-2229-44e9-b85f-279a9092dece-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.161542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.161347 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b7dce9-2229-44e9-b85f-279a9092dece-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.161542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.161362 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvvt\" (UniqueName: \"kubernetes.io/projected/67b7dce9-2229-44e9-b85f-279a9092dece-kube-api-access-mmvvt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:06.198984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.198812 5815 scope.go:122] "RemoveContainer" containerID="3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251" Nov 03 20:29:06.199191 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:29:06.199166 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251\": container with ID starting with 3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251 not found: ID does not exist" containerID="3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251" Nov 03 20:29:06.199283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.199199 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251"} err="failed to get container status \"3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251\": rpc error: code = NotFound desc = could not find container \"3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251\": container with ID starting with 3a0c58ad2919ae7c757a6b9cb41145078b858cb1d788a14d1fbaa5672812f251 not found: ID does not exist" Nov 03 20:29:06.219099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.219072 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv"] Nov 03 20:29:06.225559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.225537 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-j86hv"] Nov 03 20:29:06.602474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:06.602439 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" path="/var/lib/kubelet/pods/67b7dce9-2229-44e9-b85f-279a9092dece/volumes" Nov 03 20:29:12.352669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:12.352630 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:29:12.353158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:12.352695 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:29:13.330824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:13.330788 5815 generic.go:356] "Generic (PLEG): container finished" podID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerID="b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b" exitCode=0 Nov 03 20:29:13.331042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:13.330867 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" event={"ID":"4fb00ef0-8914-4629-a6fe-bf7eee0b3688","Type":"ContainerDied","Data":"b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b"} Nov 03 20:29:13.331379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:13.331363 5815 scope.go:122] "RemoveContainer" containerID="b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b" Nov 03 20:29:13.333178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:13.333147 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:29:13.333261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:13.333223 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:29:13.333313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:13.333276 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:29:13.870851 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:29:13.870804 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:29:14.342691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.342655 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" event={"ID":"4fb00ef0-8914-4629-a6fe-bf7eee0b3688","Type":"ContainerStarted","Data":"47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820"} Nov 03 20:29:14.594593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.584881 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:29:14.594593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.585008 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:29:14.594593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.585065 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:29:14.847116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.847047 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6"] Nov 03 20:29:14.848119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.848102 5815 cpu_manager.go:414] "containerMap: removing container" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" Nov 03 20:29:14.848253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.848243 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" Nov 03 20:29:14.848519 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.848493 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b7dce9-2229-44e9-b85f-279a9092dece" containerName="istio-proxy" Nov 03 20:29:14.859208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.859182 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.885539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.866944 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ba89e0-openshift-default-dockercfg-rvpjm\"" Nov 03 20:29:14.915986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.915935 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6"] Nov 03 20:29:14.923211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.923178 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.923447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.923425 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frnb8\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.923761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.923738 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.923947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.923925 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.924101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.924084 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.924308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.924288 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.924476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.924459 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.924723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.924704 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:14.924885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:14.924851 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.026794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.026757 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.026976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.026813 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.026976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.026851 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.026976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.026926 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.026976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.026971 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027054 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027086 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027170 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027209 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frnb8\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027264 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027318 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027356 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027541 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.027873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.027850 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.029592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.029569 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.029793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.029772 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.048970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.048947 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.056096 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.056060 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frnb8\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.242529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.234648 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:15.591591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.590940 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:29:15.591591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.591010 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:29:15.641177 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:29:15.641140 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1f1ccc_1dca_4bbc_8eef_864382955a90.slice/crio-dffe0d3b7129e6ab42bf4763a507af7a603647e6d2790e9da614892c720c8cbc WatchSource:0}: Error finding container dffe0d3b7129e6ab42bf4763a507af7a603647e6d2790e9da614892c720c8cbc: Status 404 returned error can't find the container with id dffe0d3b7129e6ab42bf4763a507af7a603647e6d2790e9da614892c720c8cbc Nov 03 20:29:15.642281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.642257 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6"] Nov 03 20:29:15.644003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.643972 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:29:15.644091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.644036 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:29:15.644091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:15.644063 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:29:16.381168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:16.381124 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" event={"ID":"2f1f1ccc-1dca-4bbc-8eef-864382955a90","Type":"ContainerStarted","Data":"7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab"} Nov 03 20:29:16.381756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:16.381199 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" event={"ID":"2f1f1ccc-1dca-4bbc-8eef-864382955a90","Type":"ContainerStarted","Data":"dffe0d3b7129e6ab42bf4763a507af7a603647e6d2790e9da614892c720c8cbc"} Nov 03 20:29:16.433377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:16.433322 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podStartSLOduration=2.433302768 podStartE2EDuration="2.433302768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:16.428953305 +0000 UTC m=+1665.959299003" watchObservedRunningTime="2026-11-03 20:29:16.433302768 +0000 UTC m=+1665.963648441" Nov 03 20:29:16.587533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:16.586924 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:29:16.587533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:16.586986 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:29:17.236079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:17.235897 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:17.237122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:17.237097 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 20:29:17.237233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:17.237157 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 20:29:17.582766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:17.582669 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:29:17.583536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:17.583473 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:29:18.241539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:18.240723 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 20:29:18.241539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:18.240785 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 20:29:18.605588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:18.604652 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:29:18.605588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:18.604712 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:29:19.236722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:19.236686 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 20:29:19.236929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:19.236734 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 20:29:19.583333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:19.583246 5815 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 03 20:29:19.583622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:19.583593 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 03 20:29:20.235667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:20.235629 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 20:29:20.236175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:20.235695 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 20:29:20.679595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:20.666669 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:29:20.679595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:20.672111 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:29:20.698545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:20.696822 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:29:21.236109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:21.236068 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 20:29:21.236589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:21.236165 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 20:29:22.237660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:22.237626 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body= Nov 03 20:29:22.257885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:22.257845 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" Nov 03 20:29:23.051291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:23.051252 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp"] Nov 03 20:29:23.062596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:23.062558 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837a9a5a-tvbnp"] Nov 03 20:29:23.240717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:23.240695 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:23.241162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:23.241036 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:23.241801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:23.241782 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 20:29:24.623854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:24.623820 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f6f54ca-2615-4dad-b55c-bb03e957ff47" path="/var/lib/kubelet/pods/9f6f54ca-2615-4dad-b55c-bb03e957ff47/volumes" Nov 03 20:29:35.952257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:35.952219 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:29:36.035363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:36.035325 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn"] Nov 03 20:29:36.041084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:36.041057 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2409989a-t62fn"] Nov 03 20:29:36.589344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:36.589310 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b27accbd-71f6-441a-bc62-9f4e1aa53f4b" path="/var/lib/kubelet/pods/b27accbd-71f6-441a-bc62-9f4e1aa53f4b/volumes" Nov 03 20:29:39.027745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:39.027713 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7"] Nov 03 20:29:39.032936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:39.032911 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15094ed4-qqcx7"] Nov 03 20:29:40.589057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:40.589022 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b4351ba-b1cb-467d-ab0b-7158f7164dcc" path="/var/lib/kubelet/pods/1b4351ba-b1cb-467d-ab0b-7158f7164dcc/volumes" Nov 03 20:29:43.029168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:43.029135 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9"] Nov 03 20:29:43.034423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:43.034396 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd4b92f-q8ck9"] Nov 03 20:29:43.855205 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:29:43.855173 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:29:44.590452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:44.590421 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e67380-5590-447a-a89b-02f673af0c8d" path="/var/lib/kubelet/pods/02e67380-5590-447a-a89b-02f673af0c8d/volumes" Nov 03 20:29:50.035792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:50.035759 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd"] Nov 03 20:29:50.041050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:50.041024 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63693ee7-7gmnd"] Nov 03 20:29:50.620419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:50.620380 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813a84c2-5db4-4537-b612-c14de26429bd" path="/var/lib/kubelet/pods/813a84c2-5db4-4537-b612-c14de26429bd/volumes" Nov 03 20:29:53.026163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:53.026127 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc"] Nov 03 20:29:53.031228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:53.031205 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de05a226-wkdpc"] Nov 03 20:29:54.026430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:54.026396 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb"] Nov 03 20:29:54.031971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:54.031942 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793d11d7-f6nkb"] Nov 03 20:29:54.591842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:54.591797 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cf1080-ad76-4f85-8cba-033b0e254267" path="/var/lib/kubelet/pods/14cf1080-ad76-4f85-8cba-033b0e254267/volumes" Nov 03 20:29:54.592574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:54.592556 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7dad49-c112-4215-95ba-9fbfe5d911c0" path="/var/lib/kubelet/pods/ab7dad49-c112-4215-95ba-9fbfe5d911c0/volumes" Nov 03 20:29:59.116528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:59.109563 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65"] Nov 03 20:29:59.134528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:29:59.130718 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d89d48f-76r65"] Nov 03 20:30:00.169203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.169164 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl"] Nov 03 20:30:00.176408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.176346 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.177162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.177140 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh"] Nov 03 20:30:00.184106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.182979 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.198424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.186595 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2"] Nov 03 20:30:00.198424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.187356 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:30:00.208389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.208357 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnx4\" (UniqueName: \"kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.208792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.208768 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.224729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.224702 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6"] Nov 03 20:30:00.225321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.225303 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.238396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.238369 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht"] Nov 03 20:30:00.271659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.239480 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.271659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.250855 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh"] Nov 03 20:30:00.271659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.250974 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.280493 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.280464 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl"] Nov 03 20:30:00.284116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.284064 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6"] Nov 03 20:30:00.291741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.291716 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht"] Nov 03 20:30:00.297610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.297589 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78"] Nov 03 20:30:00.304113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.304094 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.305225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.305210 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55"] Nov 03 20:30:00.310617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.310593 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.310838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.310819 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.311092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.311072 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhf7\" (UniqueName: \"kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.311233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.311219 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cs8\" (UniqueName: \"kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8\") pod \"maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.311364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.311349 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.311529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.311496 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.311863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.311845 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnx4\" (UniqueName: \"kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.312150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.312133 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v656b\" (UniqueName: \"kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.312302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.312287 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.312636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.312564 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.313829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.313529 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.320159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.320138 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4"] Nov 03 20:30:00.320376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.320359 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.326188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.326120 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.328441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.328426 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnx4\" (UniqueName: \"kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.328829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.328702 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78"] Nov 03 20:30:00.333435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.333384 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55"] Nov 03 20:30:00.338148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.338132 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4"] Nov 03 20:30:00.342907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.342885 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22"] Nov 03 20:30:00.347649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.347627 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2"] Nov 03 20:30:00.347743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.347697 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.352281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.352264 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77"] Nov 03 20:30:00.357079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.357062 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.362442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.362427 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22"] Nov 03 20:30:00.367326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.367309 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77"] Nov 03 20:30:00.414636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414607 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-2kg77\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.414833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414660 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mh9b\" (UniqueName: \"kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.414833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414684 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krv9\" (UniqueName: \"kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.414833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414770 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.415019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.415019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414943 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtblr\" (UniqueName: \"kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr\") pod \"maas-api-key-cleanup-29895630-2kg77\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.415019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.414999 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxj59\" (UniqueName: \"kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59\") pod \"maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.415193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.415062 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.415193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.415135 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhf7\" (UniqueName: \"kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.415193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.415174 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cs8\" (UniqueName: \"kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8\") pod \"maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.415411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.415206 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.415654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.415625 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.415791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.415771 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.416136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416118 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.416269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416250 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.416314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416293 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.416357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416333 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.416409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416389 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq6j\" (UniqueName: \"kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.416565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416529 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v656b\" (UniqueName: \"kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.416565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416561 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.416715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416584 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.416871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.416853 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.428206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.428157 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cs8\" (UniqueName: \"kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8\") pod \"maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.428301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.428209 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhf7\" (UniqueName: \"kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.428772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.428751 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.428885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.428870 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v656b\" (UniqueName: \"kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.518558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518477 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxj59\" (UniqueName: \"kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59\") pod \"maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.518701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518673 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.518807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518714 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.518807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518775 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq6j\" (UniqueName: \"kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.518954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518846 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.518954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518878 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-2kg77\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.518954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518934 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mh9b\" (UniqueName: \"kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.519137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518967 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krv9\" (UniqueName: \"kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.519137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.518995 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.519137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.519129 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtblr\" (UniqueName: \"kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr\") pod \"maas-api-key-cleanup-29895630-2kg77\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.519475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.519452 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.519627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.519605 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.519689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.519670 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.519689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.519682 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895630-2kg77\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.519895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.519878 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.523946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.523929 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:00.531889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.531863 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtblr\" (UniqueName: \"kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr\") pod \"maas-api-key-cleanup-29895630-2kg77\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:00.532009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.531867 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq6j\" (UniqueName: \"kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.532083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.532003 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxj59\" (UniqueName: \"kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59\") pod \"maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.532294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.532271 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mh9b\" (UniqueName: \"kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.532355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.532296 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krv9\" (UniqueName: \"kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.538150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.538127 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:00.591308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.591259 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:00.601911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.601876 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e465645b-7a0f-47b4-b52e-ff005b316b1b" path="/var/lib/kubelet/pods/e465645b-7a0f-47b4-b52e-ff005b316b1b/volumes" Nov 03 20:30:00.632532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.629474 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:00.745613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.704843 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:00.745613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.737265 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:00.745613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.742133 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:00.750403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.748218 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:00.750403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.748329 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:00.750403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:00.748403 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:01.073343 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:30:01.068750 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f13de40_0a6a_443c_a0ec_0c3c07f5e1c8.slice/crio-7ba7ebf510891f4a5f725904197b73e22e0741b0c0e51e250d5e058c19f4d09e WatchSource:0}: Error finding container 7ba7ebf510891f4a5f725904197b73e22e0741b0c0e51e250d5e058c19f4d09e: Status 404 returned error can't find the container with id 7ba7ebf510891f4a5f725904197b73e22e0741b0c0e51e250d5e058c19f4d09e Nov 03 20:30:01.087016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.083440 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl"] Nov 03 20:30:01.127298 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:30:01.127271 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98a3c1e_2c39_4eb2_9966_4d72a8b2192c.slice/crio-6c294ac086774b2137233232247d82e446ff80c512a24038c3a50079ecc27e39 WatchSource:0}: Error finding container 6c294ac086774b2137233232247d82e446ff80c512a24038c3a50079ecc27e39: Status 404 returned error can't find the container with id 6c294ac086774b2137233232247d82e446ff80c512a24038c3a50079ecc27e39 Nov 03 20:30:01.128611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.128583 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh"] Nov 03 20:30:01.172910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.172351 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerStarted","Data":"6c294ac086774b2137233232247d82e446ff80c512a24038c3a50079ecc27e39"} Nov 03 20:30:01.174755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.174718 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerStarted","Data":"7ba7ebf510891f4a5f725904197b73e22e0741b0c0e51e250d5e058c19f4d09e"} Nov 03 20:30:01.249166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.249139 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2"] Nov 03 20:30:01.311306 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:30:01.311281 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa298ac_ad45_4159_814e_5f339b7e9c01.slice/crio-555c9e218a9c887100546241bc1c63d924983cfdcbd6ade11cd438cf0e925dd5 WatchSource:0}: Error finding container 555c9e218a9c887100546241bc1c63d924983cfdcbd6ade11cd438cf0e925dd5: Status 404 returned error can't find the container with id 555c9e218a9c887100546241bc1c63d924983cfdcbd6ade11cd438cf0e925dd5 Nov 03 20:30:01.453541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.452124 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6"] Nov 03 20:30:01.483626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.476799 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht"] Nov 03 20:30:01.528766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.528563 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77"] Nov 03 20:30:01.536939 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:30:01.536913 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ed922e_05c1_4bf1_a826_593f11758f04.slice/crio-e1fc066576ff7bf1c8338c56b45c4847555258e4cd15877f697dbf545471550a WatchSource:0}: Error finding container e1fc066576ff7bf1c8338c56b45c4847555258e4cd15877f697dbf545471550a: Status 404 returned error can't find the container with id e1fc066576ff7bf1c8338c56b45c4847555258e4cd15877f697dbf545471550a Nov 03 20:30:01.558381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.541675 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22"] Nov 03 20:30:01.558381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.555365 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55"] Nov 03 20:30:01.561888 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:30:01.561864 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf3353e_bc17_4267_b45b_79f0b4d13352.slice/crio-0b4c371f0d8921f18e9d9c3d0b180707f704487ebf91f105ef314c634202caa2 WatchSource:0}: Error finding container 0b4c371f0d8921f18e9d9c3d0b180707f704487ebf91f105ef314c634202caa2: Status 404 returned error can't find the container with id 0b4c371f0d8921f18e9d9c3d0b180707f704487ebf91f105ef314c634202caa2 Nov 03 20:30:01.562227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.562058 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78"] Nov 03 20:30:01.574075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:01.573971 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4"] Nov 03 20:30:01.577547 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:30:01.577495 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde1671e_7d58_4a8d_b2a0_e809c8fe66e6.slice/crio-a0cbd45925d86713947dfdb53b439cd9a719d132d454155709c136d180b3c0f5 WatchSource:0}: Error finding container a0cbd45925d86713947dfdb53b439cd9a719d132d454155709c136d180b3c0f5: Status 404 returned error can't find the container with id a0cbd45925d86713947dfdb53b439cd9a719d132d454155709c136d180b3c0f5 Nov 03 20:30:02.261677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.261000 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerStarted","Data":"f40ce2109eb002a8914d193423e6c9af70579267fd43929d883006f7814b266b"} Nov 03 20:30:02.273196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.272202 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerStarted","Data":"b2e5a280463cafb14f05563be2a17b5e3593d2f4f9c178fcb2ae1535eff73059"} Nov 03 20:30:02.279861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.279795 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerStarted","Data":"0b4c371f0d8921f18e9d9c3d0b180707f704487ebf91f105ef314c634202caa2"} Nov 03 20:30:02.283955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.283895 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerStarted","Data":"ddc78a3d79b565fb8323c690141c4cf2ce7032cef3ed6b4da855313967466096"} Nov 03 20:30:02.286623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.286119 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerStarted","Data":"9267f005748b899b34d8fe51ed527b79f1c76a700400379b3a87d50e0617e4c0"} Nov 03 20:30:02.296375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.295401 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerStarted","Data":"84a314c4793da715fa4a7e749da930de7284ad1d86799ddf8f78b741f1627710"} Nov 03 20:30:02.296375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.295448 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerStarted","Data":"e1fc066576ff7bf1c8338c56b45c4847555258e4cd15877f697dbf545471550a"} Nov 03 20:30:02.301588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.301442 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" podStartSLOduration=2.301425475 podStartE2EDuration="2.301425475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.28646867 +0000 UTC m=+1711.816814343" watchObservedRunningTime="2026-11-03 20:30:02.301425475 +0000 UTC m=+1711.831771148" Nov 03 20:30:02.301983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.301957 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" event={"ID":"3fa298ac-ad45-4159-814e-5f339b7e9c01","Type":"ContainerStarted","Data":"555c9e218a9c887100546241bc1c63d924983cfdcbd6ade11cd438cf0e925dd5"} Nov 03 20:30:02.308193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.307419 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerStarted","Data":"ad845203b837befacbcbebfc232e291ccf2fe1114fc6b3144e706526c89aecaf"} Nov 03 20:30:02.308193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.307465 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerStarted","Data":"13391665e2474da2497026d64a816152c677f45b1673e881cb63391302dd80f8"} Nov 03 20:30:02.310357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.309540 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" podStartSLOduration=2.309525159 podStartE2EDuration="2.309525159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.304905142 +0000 UTC m=+1711.835250833" watchObservedRunningTime="2026-11-03 20:30:02.309525159 +0000 UTC m=+1711.839870830" Nov 03 20:30:02.314849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.314803 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerStarted","Data":"a0cbd45925d86713947dfdb53b439cd9a719d132d454155709c136d180b3c0f5"} Nov 03 20:30:02.317058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.317036 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerStarted","Data":"d9c5ffe02019b144427dded5670e98796f923e4f74fa250f8f1e605f5a10f1f7"} Nov 03 20:30:02.330044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.329978 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" podStartSLOduration=2.329965402 podStartE2EDuration="2.329965402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.327219193 +0000 UTC m=+1711.857564877" watchObservedRunningTime="2026-11-03 20:30:02.329965402 +0000 UTC m=+1711.860311074" Nov 03 20:30:02.354938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:02.354899 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" podStartSLOduration=2.3548861309999998 podStartE2EDuration="2.354886131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.34991252 +0000 UTC m=+1711.880258198" watchObservedRunningTime="2026-11-03 20:30:02.354886131 +0000 UTC m=+1711.885231803" Nov 03 20:30:03.330336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.330299 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerID="3caf2b21d9818084c120ab1b68f8c88e1838b90eba1e8611d285c199816c068b" exitCode=60 Nov 03 20:30:03.330783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.330399 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerDied","Data":"3caf2b21d9818084c120ab1b68f8c88e1838b90eba1e8611d285c199816c068b"} Nov 03 20:30:03.330833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.330785 5815 scope.go:122] "RemoveContainer" containerID="3caf2b21d9818084c120ab1b68f8c88e1838b90eba1e8611d285c199816c068b" Nov 03 20:30:03.332415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.332388 5815 generic.go:356] "Generic (PLEG): container finished" podID="4ac25daf-4025-40c5-b675-368805684a43" containerID="1a9f1680f25eef56f012fe7e845c177e699b68217661b924615ec2473ce3ee9f" exitCode=60 Nov 03 20:30:03.332587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.332478 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerDied","Data":"1a9f1680f25eef56f012fe7e845c177e699b68217661b924615ec2473ce3ee9f"} Nov 03 20:30:03.332838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.332823 5815 scope.go:122] "RemoveContainer" containerID="1a9f1680f25eef56f012fe7e845c177e699b68217661b924615ec2473ce3ee9f" Nov 03 20:30:03.334654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.334620 5815 generic.go:356] "Generic (PLEG): container finished" podID="07ed922e-05c1-4bf1-a826-593f11758f04" containerID="84a314c4793da715fa4a7e749da930de7284ad1d86799ddf8f78b741f1627710" exitCode=60 Nov 03 20:30:03.334773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.334742 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerDied","Data":"84a314c4793da715fa4a7e749da930de7284ad1d86799ddf8f78b741f1627710"} Nov 03 20:30:03.335082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.335014 5815 scope.go:122] "RemoveContainer" containerID="84a314c4793da715fa4a7e749da930de7284ad1d86799ddf8f78b741f1627710" Nov 03 20:30:03.336639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.336614 5815 generic.go:356] "Generic (PLEG): container finished" podID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerID="5df726f7ea002e9f8a7ebaceb76ded9faf749ab393be8c49635e7ec5f62877c9" exitCode=60 Nov 03 20:30:03.337111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.337100 5815 scope.go:122] "RemoveContainer" containerID="5df726f7ea002e9f8a7ebaceb76ded9faf749ab393be8c49635e7ec5f62877c9" Nov 03 20:30:03.337413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.337392 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" event={"ID":"3fa298ac-ad45-4159-814e-5f339b7e9c01","Type":"ContainerDied","Data":"5df726f7ea002e9f8a7ebaceb76ded9faf749ab393be8c49635e7ec5f62877c9"} Nov 03 20:30:03.339407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.339388 5815 generic.go:356] "Generic (PLEG): container finished" podID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerID="ad845203b837befacbcbebfc232e291ccf2fe1114fc6b3144e706526c89aecaf" exitCode=60 Nov 03 20:30:03.339538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.339436 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerDied","Data":"ad845203b837befacbcbebfc232e291ccf2fe1114fc6b3144e706526c89aecaf"} Nov 03 20:30:03.339820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.339802 5815 scope.go:122] "RemoveContainer" containerID="ad845203b837befacbcbebfc232e291ccf2fe1114fc6b3144e706526c89aecaf" Nov 03 20:30:03.341481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.341462 5815 generic.go:356] "Generic (PLEG): container finished" podID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerID="b5a89373d89353a9fd02a8a0175c9a27f8e1d60cebd11b6f28dc0c58796c47cb" exitCode=60 Nov 03 20:30:03.341770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.341755 5815 scope.go:122] "RemoveContainer" containerID="b5a89373d89353a9fd02a8a0175c9a27f8e1d60cebd11b6f28dc0c58796c47cb" Nov 03 20:30:03.341852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.341818 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerDied","Data":"b5a89373d89353a9fd02a8a0175c9a27f8e1d60cebd11b6f28dc0c58796c47cb"} Nov 03 20:30:03.343451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.343434 5815 generic.go:356] "Generic (PLEG): container finished" podID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerID="225e9371b000cf3d80bcd5491a4276b828faa70bfad7afd9a7dc54bbd8167854" exitCode=60 Nov 03 20:30:03.343584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.343520 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerDied","Data":"225e9371b000cf3d80bcd5491a4276b828faa70bfad7afd9a7dc54bbd8167854"} Nov 03 20:30:03.343747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.343731 5815 scope.go:122] "RemoveContainer" containerID="225e9371b000cf3d80bcd5491a4276b828faa70bfad7afd9a7dc54bbd8167854" Nov 03 20:30:03.345407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.345391 5815 generic.go:356] "Generic (PLEG): container finished" podID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerID="f40ce2109eb002a8914d193423e6c9af70579267fd43929d883006f7814b266b" exitCode=60 Nov 03 20:30:03.345539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.345461 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerDied","Data":"f40ce2109eb002a8914d193423e6c9af70579267fd43929d883006f7814b266b"} Nov 03 20:30:03.345746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.345724 5815 scope.go:122] "RemoveContainer" containerID="f40ce2109eb002a8914d193423e6c9af70579267fd43929d883006f7814b266b" Nov 03 20:30:03.348697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.348102 5815 generic.go:356] "Generic (PLEG): container finished" podID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerID="b2e5a280463cafb14f05563be2a17b5e3593d2f4f9c178fcb2ae1535eff73059" exitCode=60 Nov 03 20:30:03.348697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.348148 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerDied","Data":"b2e5a280463cafb14f05563be2a17b5e3593d2f4f9c178fcb2ae1535eff73059"} Nov 03 20:30:03.348697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.348422 5815 scope.go:122] "RemoveContainer" containerID="b2e5a280463cafb14f05563be2a17b5e3593d2f4f9c178fcb2ae1535eff73059" Nov 03 20:30:03.351091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.351071 5815 generic.go:356] "Generic (PLEG): container finished" podID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerID="ebefb3cb9bc8e6c38829085f3de0b825783dfb2f626b13f6ea3d67008bcd6b5b" exitCode=60 Nov 03 20:30:03.351213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.351123 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerDied","Data":"ebefb3cb9bc8e6c38829085f3de0b825783dfb2f626b13f6ea3d67008bcd6b5b"} Nov 03 20:30:03.351381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:03.351365 5815 scope.go:122] "RemoveContainer" containerID="ebefb3cb9bc8e6c38829085f3de0b825783dfb2f626b13f6ea3d67008bcd6b5b" Nov 03 20:30:04.363377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.363343 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" event={"ID":"3fa298ac-ad45-4159-814e-5f339b7e9c01","Type":"ContainerStarted","Data":"5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572"} Nov 03 20:30:04.373774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.373735 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerStarted","Data":"413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02"} Nov 03 20:30:04.377251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.377221 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerStarted","Data":"b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50"} Nov 03 20:30:04.379745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.379720 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerStarted","Data":"13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96"} Nov 03 20:30:04.382063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.382038 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerStarted","Data":"41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376"} Nov 03 20:30:04.384554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.384462 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" podStartSLOduration=4.384447621 podStartE2EDuration="4.384447621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.384116712 +0000 UTC m=+1713.914462411" watchObservedRunningTime="2026-11-03 20:30:04.384447621 +0000 UTC m=+1713.914793294" Nov 03 20:30:04.385103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.385083 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerStarted","Data":"232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f"} Nov 03 20:30:04.387299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.387274 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerStarted","Data":"1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2"} Nov 03 20:30:04.389863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.389841 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerStarted","Data":"a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414"} Nov 03 20:30:04.393276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.393252 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerStarted","Data":"8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47"} Nov 03 20:30:04.395438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.395417 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerStarted","Data":"e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847"} Nov 03 20:30:04.405248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.405204 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" podStartSLOduration=4.405190898 podStartE2EDuration="4.405190898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.401782851 +0000 UTC m=+1713.932128524" watchObservedRunningTime="2026-11-03 20:30:04.405190898 +0000 UTC m=+1713.935536576" Nov 03 20:30:04.441677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.441589 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" podStartSLOduration=4.441572883 podStartE2EDuration="4.441572883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.437186456 +0000 UTC m=+1713.967532128" watchObservedRunningTime="2026-11-03 20:30:04.441572883 +0000 UTC m=+1713.971918559" Nov 03 20:30:04.499580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.499525 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" podStartSLOduration=4.499490105 podStartE2EDuration="4.499490105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.493897508 +0000 UTC m=+1714.024243181" watchObservedRunningTime="2026-11-03 20:30:04.499490105 +0000 UTC m=+1714.029835778" Nov 03 20:30:04.526488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.526447 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" podStartSLOduration=4.526430337 podStartE2EDuration="4.526430337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.523150969 +0000 UTC m=+1714.053496642" watchObservedRunningTime="2026-11-03 20:30:04.526430337 +0000 UTC m=+1714.056776011" Nov 03 20:30:04.559884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:04.559837 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" podStartSLOduration=4.559822758 podStartE2EDuration="4.559822758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.557745867 +0000 UTC m=+1714.088091540" watchObservedRunningTime="2026-11-03 20:30:04.559822758 +0000 UTC m=+1714.090168430" Nov 03 20:30:05.451552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.451468 5815 generic.go:356] "Generic (PLEG): container finished" podID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerID="41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376" exitCode=60 Nov 03 20:30:05.452024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.451630 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerDied","Data":"41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376"} Nov 03 20:30:05.452024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.451688 5815 scope.go:122] "RemoveContainer" containerID="f40ce2109eb002a8914d193423e6c9af70579267fd43929d883006f7814b266b" Nov 03 20:30:05.452170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.452067 5815 scope.go:122] "RemoveContainer" containerID="41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376" Nov 03 20:30:05.452614 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.452589 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh_odh-ai-gateway-infra(e98a3c1e-2c39-4eb2-9966-4d72a8b2192c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" Nov 03 20:30:05.454254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.454223 5815 generic.go:356] "Generic (PLEG): container finished" podID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerID="232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f" exitCode=60 Nov 03 20:30:05.454341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.454299 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerDied","Data":"232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f"} Nov 03 20:30:05.454851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.454835 5815 scope.go:122] "RemoveContainer" containerID="232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f" Nov 03 20:30:05.455247 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.455226 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl_odh-ai-gateway-infra(9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" Nov 03 20:30:05.457490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.457467 5815 generic.go:356] "Generic (PLEG): container finished" podID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerID="1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2" exitCode=60 Nov 03 20:30:05.457636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.457618 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerDied","Data":"1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2"} Nov 03 20:30:05.457879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.457861 5815 scope.go:122] "RemoveContainer" containerID="1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2" Nov 03 20:30:05.458230 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.458206 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78_odh-ai-gateway-infra(fdf3353e-bc17-4267-b45b-79f0b4d13352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" Nov 03 20:30:05.459901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.459881 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerID="a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414" exitCode=60 Nov 03 20:30:05.459986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.459903 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerDied","Data":"a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414"} Nov 03 20:30:05.460238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.460221 5815 scope.go:122] "RemoveContainer" containerID="a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414" Nov 03 20:30:05.460653 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.460630 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4_odh-ai-gateway-infra(9a16b908-3cf0-4e38-9cf4-8c339c642887)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" Nov 03 20:30:05.462281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.462262 5815 generic.go:356] "Generic (PLEG): container finished" podID="4ac25daf-4025-40c5-b675-368805684a43" containerID="8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47" exitCode=60 Nov 03 20:30:05.462611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.462586 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerDied","Data":"8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47"} Nov 03 20:30:05.462794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.462780 5815 scope.go:122] "RemoveContainer" containerID="8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47" Nov 03 20:30:05.463257 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.463234 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-2kg77_odh-ai-gateway-infra(4ac25daf-4025-40c5-b675-368805684a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" podUID="4ac25daf-4025-40c5-b675-368805684a43" Nov 03 20:30:05.465175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.465152 5815 generic.go:356] "Generic (PLEG): container finished" podID="07ed922e-05c1-4bf1-a826-593f11758f04" containerID="e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847" exitCode=60 Nov 03 20:30:05.465308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.465241 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerDied","Data":"e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847"} Nov 03 20:30:05.465699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.465457 5815 scope.go:122] "RemoveContainer" containerID="e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847" Nov 03 20:30:05.465796 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.465735 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55_odh-ai-gateway-infra(07ed922e-05c1-4bf1-a826-593f11758f04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" Nov 03 20:30:05.468620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.467813 5815 generic.go:356] "Generic (PLEG): container finished" podID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerID="5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572" exitCode=60 Nov 03 20:30:05.468620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.467882 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" event={"ID":"3fa298ac-ad45-4159-814e-5f339b7e9c01","Type":"ContainerDied","Data":"5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572"} Nov 03 20:30:05.468620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.468182 5815 scope.go:122] "RemoveContainer" containerID="5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572" Nov 03 20:30:05.468620 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.468428 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6_odh-ai-gateway-infra(3fa298ac-ad45-4159-814e-5f339b7e9c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" Nov 03 20:30:05.470851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.470827 5815 generic.go:356] "Generic (PLEG): container finished" podID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerID="413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02" exitCode=60 Nov 03 20:30:05.470941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.470903 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerDied","Data":"413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02"} Nov 03 20:30:05.471099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.471083 5815 scope.go:122] "RemoveContainer" containerID="413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02" Nov 03 20:30:05.471334 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.471315 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2_odh-ai-gateway-infra(3d868523-6bea-49f6-b8bc-56d9a4a44ec1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" Nov 03 20:30:05.474225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.472975 5815 scope.go:122] "RemoveContainer" containerID="b2e5a280463cafb14f05563be2a17b5e3593d2f4f9c178fcb2ae1535eff73059" Nov 03 20:30:05.474225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.473446 5815 generic.go:356] "Generic (PLEG): container finished" podID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerID="b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50" exitCode=60 Nov 03 20:30:05.474225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.473520 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerDied","Data":"b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50"} Nov 03 20:30:05.474225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.474166 5815 scope.go:122] "RemoveContainer" containerID="b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50" Nov 03 20:30:05.474494 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.474373 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22_odh-ai-gateway-infra(cde1671e-7d58-4a8d-b2a0-e809c8fe66e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" Nov 03 20:30:05.477104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.477082 5815 generic.go:356] "Generic (PLEG): container finished" podID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerID="13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96" exitCode=60 Nov 03 20:30:05.477216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.477129 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerDied","Data":"13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96"} Nov 03 20:30:05.477361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.477346 5815 scope.go:122] "RemoveContainer" containerID="13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96" Nov 03 20:30:05.477720 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:05.477698 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht_odh-ai-gateway-infra(c442837a-8c84-40ad-b101-e8e3d89087ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" Nov 03 20:30:05.490361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.490342 5815 scope.go:122] "RemoveContainer" containerID="ebefb3cb9bc8e6c38829085f3de0b825783dfb2f626b13f6ea3d67008bcd6b5b" Nov 03 20:30:05.516377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.516352 5815 scope.go:122] "RemoveContainer" containerID="3caf2b21d9818084c120ab1b68f8c88e1838b90eba1e8611d285c199816c068b" Nov 03 20:30:05.533283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.533267 5815 scope.go:122] "RemoveContainer" containerID="1a9f1680f25eef56f012fe7e845c177e699b68217661b924615ec2473ce3ee9f" Nov 03 20:30:05.548083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.548064 5815 scope.go:122] "RemoveContainer" containerID="84a314c4793da715fa4a7e749da930de7284ad1d86799ddf8f78b741f1627710" Nov 03 20:30:05.562250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.562231 5815 scope.go:122] "RemoveContainer" containerID="5df726f7ea002e9f8a7ebaceb76ded9faf749ab393be8c49635e7ec5f62877c9" Nov 03 20:30:05.578327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.578307 5815 scope.go:122] "RemoveContainer" containerID="ad845203b837befacbcbebfc232e291ccf2fe1114fc6b3144e706526c89aecaf" Nov 03 20:30:05.597654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.597634 5815 scope.go:122] "RemoveContainer" containerID="b5a89373d89353a9fd02a8a0175c9a27f8e1d60cebd11b6f28dc0c58796c47cb" Nov 03 20:30:05.616037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:05.616019 5815 scope.go:122] "RemoveContainer" containerID="225e9371b000cf3d80bcd5491a4276b828faa70bfad7afd9a7dc54bbd8167854" Nov 03 20:30:06.040095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.040054 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj"] Nov 03 20:30:06.041074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.041046 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98bd36cc-pjqjj"] Nov 03 20:30:06.492375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.492342 5815 scope.go:122] "RemoveContainer" containerID="41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376" Nov 03 20:30:06.493053 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.492812 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh_odh-ai-gateway-infra(e98a3c1e-2c39-4eb2-9966-4d72a8b2192c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" Nov 03 20:30:06.494386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.494364 5815 scope.go:122] "RemoveContainer" containerID="232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f" Nov 03 20:30:06.494822 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.494792 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl_odh-ai-gateway-infra(9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" Nov 03 20:30:06.497287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.497254 5815 scope.go:122] "RemoveContainer" containerID="1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2" Nov 03 20:30:06.497665 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.497643 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78_odh-ai-gateway-infra(fdf3353e-bc17-4267-b45b-79f0b4d13352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" Nov 03 20:30:06.499433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.499414 5815 scope.go:122] "RemoveContainer" containerID="a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414" Nov 03 20:30:06.499773 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.499743 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4_odh-ai-gateway-infra(9a16b908-3cf0-4e38-9cf4-8c339c642887)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" Nov 03 20:30:06.501321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.501303 5815 scope.go:122] "RemoveContainer" containerID="8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47" Nov 03 20:30:06.501634 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.501607 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895630-2kg77_odh-ai-gateway-infra(4ac25daf-4025-40c5-b675-368805684a43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" podUID="4ac25daf-4025-40c5-b675-368805684a43" Nov 03 20:30:06.503450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.503431 5815 scope.go:122] "RemoveContainer" containerID="e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847" Nov 03 20:30:06.503790 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.503764 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55_odh-ai-gateway-infra(07ed922e-05c1-4bf1-a826-593f11758f04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" Nov 03 20:30:06.505316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.505296 5815 scope.go:122] "RemoveContainer" containerID="5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572" Nov 03 20:30:06.505671 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.505630 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6_odh-ai-gateway-infra(3fa298ac-ad45-4159-814e-5f339b7e9c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" Nov 03 20:30:06.507265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.507246 5815 scope.go:122] "RemoveContainer" containerID="413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02" Nov 03 20:30:06.507556 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.507537 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2_odh-ai-gateway-infra(3d868523-6bea-49f6-b8bc-56d9a4a44ec1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" Nov 03 20:30:06.509123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.509103 5815 scope.go:122] "RemoveContainer" containerID="b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50" Nov 03 20:30:06.509378 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.509355 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22_odh-ai-gateway-infra(cde1671e-7d58-4a8d-b2a0-e809c8fe66e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" Nov 03 20:30:06.511921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.511903 5815 scope.go:122] "RemoveContainer" containerID="13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96" Nov 03 20:30:06.512169 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:06.512149 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht_odh-ai-gateway-infra(c442837a-8c84-40ad-b101-e8e3d89087ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" Nov 03 20:30:06.593019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:06.592981 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23e8318-e623-4dd1-ba53-87eac194fbcf" path="/var/lib/kubelet/pods/a23e8318-e623-4dd1-ba53-87eac194fbcf/volumes" Nov 03 20:30:07.026556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:07.026460 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc"] Nov 03 20:30:07.032077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:07.032048 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b378a5a0-g8xpc"] Nov 03 20:30:08.591743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:08.591671 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fb2a69-834a-47fd-91dd-95f09b5b4c5f" path="/var/lib/kubelet/pods/40fb2a69-834a-47fd-91dd-95f09b5b4c5f/volumes" Nov 03 20:30:13.853410 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:13.853372 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:30:15.029071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:15.029037 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2"] Nov 03 20:30:15.033952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:15.033927 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7ecd61-xqmd2"] Nov 03 20:30:16.612531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:16.608213 5815 scope.go:122] "RemoveContainer" containerID="41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376" Nov 03 20:30:16.612531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:16.609474 5815 scope.go:122] "RemoveContainer" containerID="13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96" Nov 03 20:30:16.647015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:16.640075 5815 scope.go:122] "RemoveContainer" containerID="b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50" Nov 03 20:30:16.699017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:16.698983 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3330058a-cc34-4507-b075-69b042759aff" path="/var/lib/kubelet/pods/3330058a-cc34-4507-b075-69b042759aff/volumes" Nov 03 20:30:17.578437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.578358 5815 scope.go:122] "RemoveContainer" containerID="e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847" Nov 03 20:30:17.578707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.578668 5815 scope.go:122] "RemoveContainer" containerID="5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572" Nov 03 20:30:17.578872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.578785 5815 scope.go:122] "RemoveContainer" containerID="413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02" Nov 03 20:30:17.578953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.578931 5815 scope.go:122] "RemoveContainer" containerID="8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47" Nov 03 20:30:17.579124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.579099 5815 scope.go:122] "RemoveContainer" containerID="232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f" Nov 03 20:30:17.579434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.579317 5815 scope.go:122] "RemoveContainer" containerID="a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414" Nov 03 20:30:17.721674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.721650 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerStarted","Data":"eca3d7210e2574e50a88029c6d095013b4681a1a9e26047c0054153c77d26f74"} Nov 03 20:30:17.723829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.723801 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerStarted","Data":"335d12770ba0dd6aead46ddc1a56f16feb4b82ccb5f93384e7fe8277c24194ad"} Nov 03 20:30:17.725977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:17.725952 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerStarted","Data":"43d878f8f1a3894cb8ec6b5f8471535e636cc68db939f48fc83ae0a46a55e197"} Nov 03 20:30:18.578701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.578658 5815 scope.go:122] "RemoveContainer" containerID="1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2" Nov 03 20:30:18.740888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.740859 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerStarted","Data":"50eceda07a44063e74304c0661432893d708f471767d7df4b525d1c850f48b3d"} Nov 03 20:30:18.745286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.745219 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerStarted","Data":"9ba6848538797de1b0a9fe94d1bb8fb642e3d7433799cf31fa54008e7ce5f29e"} Nov 03 20:30:18.750702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.750674 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerStarted","Data":"c20f57fc6ed03c027181410eb88ddef84904305be3b588026b46e4016a798e27"} Nov 03 20:30:18.756039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.756019 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22"] Nov 03 20:30:18.756769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.756742 5815 generic.go:356] "Generic (PLEG): container finished" podID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerID="20f82b28e777b7feb014104a01b8f7f8bad21210e0e3739f6a8bc4c219c7cd9b" exitCode=60 Nov 03 20:30:18.756868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.756819 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" event={"ID":"3fa298ac-ad45-4159-814e-5f339b7e9c01","Type":"ContainerDied","Data":"20f82b28e777b7feb014104a01b8f7f8bad21210e0e3739f6a8bc4c219c7cd9b"} Nov 03 20:30:18.756956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.756879 5815 scope.go:122] "RemoveContainer" containerID="5cfe5cb4fd28832253241c51aaf21ac8c739a35b5fe5b1b5d8021639641c8572" Nov 03 20:30:18.760250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.757344 5815 scope.go:122] "RemoveContainer" containerID="20f82b28e777b7feb014104a01b8f7f8bad21210e0e3739f6a8bc4c219c7cd9b" Nov 03 20:30:18.760250 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:18.757786 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6_odh-ai-gateway-infra(3fa298ac-ad45-4159-814e-5f339b7e9c01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" Nov 03 20:30:18.771457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.770072 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerStarted","Data":"61f4b03d92219ec39369262061a8e0390adfd54e63b3169d905a5962980b7fee"} Nov 03 20:30:18.773816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.773549 5815 generic.go:356] "Generic (PLEG): container finished" podID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerID="eca3d7210e2574e50a88029c6d095013b4681a1a9e26047c0054153c77d26f74" exitCode=60 Nov 03 20:30:18.773816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.773626 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerDied","Data":"eca3d7210e2574e50a88029c6d095013b4681a1a9e26047c0054153c77d26f74"} Nov 03 20:30:18.799842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.799818 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh"] Nov 03 20:30:18.820042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.820005 5815 generic.go:356] "Generic (PLEG): container finished" podID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerID="335d12770ba0dd6aead46ddc1a56f16feb4b82ccb5f93384e7fe8277c24194ad" exitCode=60 Nov 03 20:30:18.820273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.820255 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerDied","Data":"335d12770ba0dd6aead46ddc1a56f16feb4b82ccb5f93384e7fe8277c24194ad"} Nov 03 20:30:18.820815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.820794 5815 scope.go:122] "RemoveContainer" containerID="335d12770ba0dd6aead46ddc1a56f16feb4b82ccb5f93384e7fe8277c24194ad" Nov 03 20:30:18.821282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.821087 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht"] Nov 03 20:30:18.821282 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:18.821258 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht_odh-ai-gateway-infra(c442837a-8c84-40ad-b101-e8e3d89087ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" Nov 03 20:30:18.822659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.822636 5815 scope.go:122] "RemoveContainer" containerID="b584800ea959976d7e4af0810dd15d781bc4f7f7654b4c35632f181d9cdf9b50" Nov 03 20:30:18.823288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.823268 5815 generic.go:356] "Generic (PLEG): container finished" podID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerID="43d878f8f1a3894cb8ec6b5f8471535e636cc68db939f48fc83ae0a46a55e197" exitCode=60 Nov 03 20:30:18.823396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.823350 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerDied","Data":"43d878f8f1a3894cb8ec6b5f8471535e636cc68db939f48fc83ae0a46a55e197"} Nov 03 20:30:18.826489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.826470 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerStarted","Data":"db8571b12f9f31abd4c8ef4201f92e72b617af24efd9ec11e5ec8367e0fd3789"} Nov 03 20:30:18.841804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.841789 5815 scope.go:122] "RemoveContainer" containerID="13f04d9b29a80ed117c6d37ee120d04c760128623c13a00279ba0413abfb2a96" Nov 03 20:30:18.857850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:18.857836 5815 scope.go:122] "RemoveContainer" containerID="41c2d6419ff26800a802cc0c6121bad33476bb5b2622db6a1e6c506d9fbea376" Nov 03 20:30:19.259576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.259544 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:19.310320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.310290 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:19.397123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.397095 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca\") pod \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " Nov 03 20:30:19.397291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.397156 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca\") pod \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " Nov 03 20:30:19.397354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.397312 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7\" (UniqueName: \"kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7\") pod \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\" (UID: \"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c\") " Nov 03 20:30:19.397615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.397591 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca" pod "cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" (UID: "cde1671e-7d58-4a8d-b2a0-e809c8fe66e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:19.397720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.397686 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca" pod "e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" (UID: "e98a3c1e-2c39-4eb2-9966-4d72a8b2192c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:19.397885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.397862 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b\" (UniqueName: \"kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b\") pod \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\" (UID: \"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6\") " Nov 03 20:30:19.398639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.398618 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.398715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.398642 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.400118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.400097 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7" pod "e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" (UID: "e98a3c1e-2c39-4eb2-9966-4d72a8b2192c"). InnerVolumeSpecName "kube-api-access-chhf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:19.400402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.400376 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b" pod "cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" (UID: "cde1671e-7d58-4a8d-b2a0-e809c8fe66e6"). InnerVolumeSpecName "kube-api-access-8mh9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:19.500437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.500406 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chhf7\" (UniqueName: \"kubernetes.io/projected/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c-kube-api-access-chhf7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.500591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.500442 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mh9b\" (UniqueName: \"kubernetes.io/projected/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6-kube-api-access-8mh9b\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.782984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.782950 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4"] Nov 03 20:30:19.824724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.824693 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6"] Nov 03 20:30:19.835487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.835458 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55"] Nov 03 20:30:19.843677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.843559 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" event={"ID":"cde1671e-7d58-4a8d-b2a0-e809c8fe66e6","Type":"ContainerDied","Data":"a0cbd45925d86713947dfdb53b439cd9a719d132d454155709c136d180b3c0f5"} Nov 03 20:30:19.843677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.843636 5815 scope.go:122] "RemoveContainer" containerID="eca3d7210e2574e50a88029c6d095013b4681a1a9e26047c0054153c77d26f74" Nov 03 20:30:19.846765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.843980 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22" Nov 03 20:30:19.858383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.858364 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" Nov 03 20:30:19.858498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.858361 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh" event={"ID":"e98a3c1e-2c39-4eb2-9966-4d72a8b2192c","Type":"ContainerDied","Data":"6c294ac086774b2137233232247d82e446ff80c512a24038c3a50079ecc27e39"} Nov 03 20:30:19.862827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.862801 5815 generic.go:356] "Generic (PLEG): container finished" podID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerID="db8571b12f9f31abd4c8ef4201f92e72b617af24efd9ec11e5ec8367e0fd3789" exitCode=60 Nov 03 20:30:19.863035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.863016 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerDied","Data":"db8571b12f9f31abd4c8ef4201f92e72b617af24efd9ec11e5ec8367e0fd3789"} Nov 03 20:30:19.880083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.868577 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerID="50eceda07a44063e74304c0661432893d708f471767d7df4b525d1c850f48b3d" exitCode=60 Nov 03 20:30:19.881954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.881932 5815 scope.go:122] "RemoveContainer" containerID="db8571b12f9f31abd4c8ef4201f92e72b617af24efd9ec11e5ec8367e0fd3789" Nov 03 20:30:19.883694 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:19.883654 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl_odh-ai-gateway-infra(9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" Nov 03 20:30:19.886670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.879974 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerStarted","Data":"ddd65aa6d32f8744393ba82cd144b4939b5b977f0b36d377d821583391a4cbff"} Nov 03 20:30:19.886876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.886855 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77"] Nov 03 20:30:19.887047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.887029 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerDied","Data":"50eceda07a44063e74304c0661432893d708f471767d7df4b525d1c850f48b3d"} Nov 03 20:30:19.894358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.894300 5815 generic.go:356] "Generic (PLEG): container finished" podID="4ac25daf-4025-40c5-b675-368805684a43" containerID="9ba6848538797de1b0a9fe94d1bb8fb642e3d7433799cf31fa54008e7ce5f29e" exitCode=60 Nov 03 20:30:19.901733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.896725 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerDied","Data":"9ba6848538797de1b0a9fe94d1bb8fb642e3d7433799cf31fa54008e7ce5f29e"} Nov 03 20:30:19.907390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.907362 5815 generic.go:356] "Generic (PLEG): container finished" podID="07ed922e-05c1-4bf1-a826-593f11758f04" containerID="c20f57fc6ed03c027181410eb88ddef84904305be3b588026b46e4016a798e27" exitCode=60 Nov 03 20:30:19.907589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.907565 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerDied","Data":"c20f57fc6ed03c027181410eb88ddef84904305be3b588026b46e4016a798e27"} Nov 03 20:30:19.921130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.921101 5815 generic.go:356] "Generic (PLEG): container finished" podID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerID="61f4b03d92219ec39369262061a8e0390adfd54e63b3169d905a5962980b7fee" exitCode=60 Nov 03 20:30:19.921260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.921201 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerDied","Data":"61f4b03d92219ec39369262061a8e0390adfd54e63b3169d905a5962980b7fee"} Nov 03 20:30:19.924604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.923708 5815 scope.go:122] "RemoveContainer" containerID="61f4b03d92219ec39369262061a8e0390adfd54e63b3169d905a5962980b7fee" Nov 03 20:30:19.924604 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:19.924200 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2_odh-ai-gateway-infra(3d868523-6bea-49f6-b8bc-56d9a4a44ec1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" Nov 03 20:30:19.927365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.927293 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2"] Nov 03 20:30:19.954619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:19.954594 5815 scope.go:122] "RemoveContainer" containerID="43d878f8f1a3894cb8ec6b5f8471535e636cc68db939f48fc83ae0a46a55e197" Nov 03 20:30:20.024085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.024027 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl"] Nov 03 20:30:20.057746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.057699 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22"] Nov 03 20:30:20.060608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.060590 5815 scope.go:122] "RemoveContainer" containerID="232ce2640b025474dc35ff5d9876edb6d0469d63bc7a8d03fe7d803f9928770f" Nov 03 20:30:20.073283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.073233 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895630-4ln22"] Nov 03 20:30:20.088204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.087887 5815 scope.go:122] "RemoveContainer" containerID="a55476ea0a65180c324c145cad54d53f52e0a5bece11d1e4562c9aecbde91414" Nov 03 20:30:20.113639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.113573 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh"] Nov 03 20:30:20.125607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.122350 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895630-nncvh"] Nov 03 20:30:20.146180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.145851 5815 scope.go:122] "RemoveContainer" containerID="8cee26b0c8b2e1fef6018149727971bbdb610f1d6f8241ac0fd5efd7c22eaa47" Nov 03 20:30:20.162492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.162469 5815 scope.go:122] "RemoveContainer" containerID="e52be21f51b654114a34e0bf5f2dc614b82cd3a85d07b48f7c1c4db9c4f04847" Nov 03 20:30:20.195832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.195730 5815 scope.go:122] "RemoveContainer" containerID="413a524a465385250ba271de305e416852093b32645f2b7f242ec83f9dd55f02" Nov 03 20:30:20.262548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.262489 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:20.338242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.338209 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:20.351358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.351332 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:20.415015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.414994 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:20.425227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.425200 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:20.426730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.426704 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca\") pod \"4ac25daf-4025-40c5-b675-368805684a43\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " Nov 03 20:30:20.426835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.426810 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca\") pod \"c442837a-8c84-40ad-b101-e8e3d89087ce\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " Nov 03 20:30:20.426928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.426912 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b\" (UniqueName: \"kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b\") pod \"c442837a-8c84-40ad-b101-e8e3d89087ce\" (UID: \"c442837a-8c84-40ad-b101-e8e3d89087ce\") " Nov 03 20:30:20.427028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.427010 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca\") pod \"3fa298ac-ad45-4159-814e-5f339b7e9c01\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " Nov 03 20:30:20.427121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.427104 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr\" (UniqueName: \"kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr\") pod \"4ac25daf-4025-40c5-b675-368805684a43\" (UID: \"4ac25daf-4025-40c5-b675-368805684a43\") " Nov 03 20:30:20.427328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.427184 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca" pod "c442837a-8c84-40ad-b101-e8e3d89087ce" (UID: "c442837a-8c84-40ad-b101-e8e3d89087ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.427443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.427346 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt\") pod \"3fa298ac-ad45-4159-814e-5f339b7e9c01\" (UID: \"3fa298ac-ad45-4159-814e-5f339b7e9c01\") " Nov 03 20:30:20.427443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.427407 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca" pod "4ac25daf-4025-40c5-b675-368805684a43" (UID: "4ac25daf-4025-40c5-b675-368805684a43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.428346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.427604 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca" pod "3fa298ac-ad45-4159-814e-5f339b7e9c01" (UID: "3fa298ac-ad45-4159-814e-5f339b7e9c01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.428346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.428183 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ac25daf-4025-40c5-b675-368805684a43-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.428346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.428207 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c442837a-8c84-40ad-b101-e8e3d89087ce-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.428346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.428222 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa298ac-ad45-4159-814e-5f339b7e9c01-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.430250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.430214 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr" pod "4ac25daf-4025-40c5-b675-368805684a43" (UID: "4ac25daf-4025-40c5-b675-368805684a43"). InnerVolumeSpecName "kube-api-access-dtblr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.430368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.430342 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt" pod "3fa298ac-ad45-4159-814e-5f339b7e9c01" (UID: "3fa298ac-ad45-4159-814e-5f339b7e9c01"). InnerVolumeSpecName "kube-api-access-r4kmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.430368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.430354 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b" pod "c442837a-8c84-40ad-b101-e8e3d89087ce" (UID: "c442837a-8c84-40ad-b101-e8e3d89087ce"). InnerVolumeSpecName "kube-api-access-v656b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.563737 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca\") pod \"07ed922e-05c1-4bf1-a826-593f11758f04\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.563839 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59\" (UniqueName: \"kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59\") pod \"9a16b908-3cf0-4e38-9cf4-8c339c642887\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.563889 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j\" (UniqueName: \"kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j\") pod \"07ed922e-05c1-4bf1-a826-593f11758f04\" (UID: \"07ed922e-05c1-4bf1-a826-593f11758f04\") " Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.564128 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca\") pod \"9a16b908-3cf0-4e38-9cf4-8c339c642887\" (UID: \"9a16b908-3cf0-4e38-9cf4-8c339c642887\") " Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.564883 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v656b\" (UniqueName: \"kubernetes.io/projected/c442837a-8c84-40ad-b101-e8e3d89087ce-kube-api-access-v656b\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.564906 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtblr\" (UniqueName: \"kubernetes.io/projected/4ac25daf-4025-40c5-b675-368805684a43-kube-api-access-dtblr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.564920 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kmt\" (UniqueName: \"kubernetes.io/projected/3fa298ac-ad45-4159-814e-5f339b7e9c01-kube-api-access-r4kmt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.565793 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca" pod "07ed922e-05c1-4bf1-a826-593f11758f04" (UID: "07ed922e-05c1-4bf1-a826-593f11758f04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.568179 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59" pod "9a16b908-3cf0-4e38-9cf4-8c339c642887" (UID: "9a16b908-3cf0-4e38-9cf4-8c339c642887"). InnerVolumeSpecName "kube-api-access-xxj59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.571476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.568860 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j" pod "07ed922e-05c1-4bf1-a826-593f11758f04" (UID: "07ed922e-05c1-4bf1-a826-593f11758f04"). InnerVolumeSpecName "kube-api-access-bnq6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.572108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.572088 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca" pod "9a16b908-3cf0-4e38-9cf4-8c339c642887" (UID: "9a16b908-3cf0-4e38-9cf4-8c339c642887"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.613080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.613045 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" path="/var/lib/kubelet/pods/cde1671e-7d58-4a8d-b2a0-e809c8fe66e6/volumes" Nov 03 20:30:20.613940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.613918 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" path="/var/lib/kubelet/pods/e98a3c1e-2c39-4eb2-9966-4d72a8b2192c/volumes" Nov 03 20:30:20.666630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.666606 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07ed922e-05c1-4bf1-a826-593f11758f04-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.666748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.666636 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxj59\" (UniqueName: \"kubernetes.io/projected/9a16b908-3cf0-4e38-9cf4-8c339c642887-kube-api-access-xxj59\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.666748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.666651 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnq6j\" (UniqueName: \"kubernetes.io/projected/07ed922e-05c1-4bf1-a826-593f11758f04-kube-api-access-bnq6j\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.666748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.666666 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a16b908-3cf0-4e38-9cf4-8c339c642887-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.932993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.932899 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" event={"ID":"9a16b908-3cf0-4e38-9cf4-8c339c642887","Type":"ContainerDied","Data":"ddc78a3d79b565fb8323c690141c4cf2ce7032cef3ed6b4da855313967466096"} Nov 03 20:30:20.932993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.932965 5815 scope.go:122] "RemoveContainer" containerID="50eceda07a44063e74304c0661432893d708f471767d7df4b525d1c850f48b3d" Nov 03 20:30:20.933556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.933099 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4" Nov 03 20:30:20.935693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.935672 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" Nov 03 20:30:20.935779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.935697 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77" event={"ID":"4ac25daf-4025-40c5-b675-368805684a43","Type":"ContainerDied","Data":"9267f005748b899b34d8fe51ed527b79f1c76a700400379b3a87d50e0617e4c0"} Nov 03 20:30:20.937665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.937645 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" Nov 03 20:30:20.937665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.937652 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55" event={"ID":"07ed922e-05c1-4bf1-a826-593f11758f04","Type":"ContainerDied","Data":"e1fc066576ff7bf1c8338c56b45c4847555258e4cd15877f697dbf545471550a"} Nov 03 20:30:20.939328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.939312 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" Nov 03 20:30:20.939442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.939339 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6" event={"ID":"3fa298ac-ad45-4159-814e-5f339b7e9c01","Type":"ContainerDied","Data":"555c9e218a9c887100546241bc1c63d924983cfdcbd6ade11cd438cf0e925dd5"} Nov 03 20:30:20.951072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.951048 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" event={"ID":"c442837a-8c84-40ad-b101-e8e3d89087ce","Type":"ContainerDied","Data":"d9c5ffe02019b144427dded5670e98796f923e4f74fa250f8f1e605f5a10f1f7"} Nov 03 20:30:20.951187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.951120 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht" Nov 03 20:30:20.958831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.958805 5815 generic.go:356] "Generic (PLEG): container finished" podID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerID="ddd65aa6d32f8744393ba82cd144b4939b5b977f0b36d377d821583391a4cbff" exitCode=60 Nov 03 20:30:20.958963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.958864 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerDied","Data":"ddd65aa6d32f8744393ba82cd144b4939b5b977f0b36d377d821583391a4cbff"} Nov 03 20:30:20.959308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.959290 5815 scope.go:122] "RemoveContainer" containerID="ddd65aa6d32f8744393ba82cd144b4939b5b977f0b36d377d821583391a4cbff" Nov 03 20:30:20.959944 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:20.959640 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78_odh-ai-gateway-infra(fdf3353e-bc17-4267-b45b-79f0b4d13352)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" Nov 03 20:30:20.988887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:20.988739 5815 scope.go:122] "RemoveContainer" containerID="9ba6848538797de1b0a9fe94d1bb8fb642e3d7433799cf31fa54008e7ce5f29e" Nov 03 20:30:21.012645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.012621 5815 scope.go:122] "RemoveContainer" containerID="c20f57fc6ed03c027181410eb88ddef84904305be3b588026b46e4016a798e27" Nov 03 20:30:21.029282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.029264 5815 scope.go:122] "RemoveContainer" containerID="20f82b28e777b7feb014104a01b8f7f8bad21210e0e3739f6a8bc4c219c7cd9b" Nov 03 20:30:21.040657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.039866 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht"] Nov 03 20:30:21.050342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.050318 5815 scope.go:122] "RemoveContainer" containerID="335d12770ba0dd6aead46ddc1a56f16feb4b82ccb5f93384e7fe8277c24194ad" Nov 03 20:30:21.050883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.050395 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895630-84wht"] Nov 03 20:30:21.081755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.081733 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55"] Nov 03 20:30:21.084856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.084834 5815 scope.go:122] "RemoveContainer" containerID="1f7aeb5ce6b39ec3de7d2a5091b1df8db14d60a7ba05358f0c2da477b944a0e2" Nov 03 20:30:21.091733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.091700 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895630-rwj55"] Nov 03 20:30:21.100431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.100410 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77"] Nov 03 20:30:21.109065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.109040 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895630-2kg77"] Nov 03 20:30:21.115591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.115567 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6"] Nov 03 20:30:21.120481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.120460 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895630-k92b6"] Nov 03 20:30:21.135560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.135538 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4"] Nov 03 20:30:21.141822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.141801 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-6bd876-29895630-mq4c4"] Nov 03 20:30:21.154244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.149357 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78"] Nov 03 20:30:21.363142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.363113 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:21.366265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.366239 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.441338 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8\" (UniqueName: \"kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8\") pod \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.441409 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca\") pod \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\" (UID: \"3d868523-6bea-49f6-b8bc-56d9a4a44ec1\") " Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.441536 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4\" (UniqueName: \"kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4\") pod \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.441585 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca\") pod \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\" (UID: \"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8\") " Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.441797 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca" pod "3d868523-6bea-49f6-b8bc-56d9a4a44ec1" (UID: "3d868523-6bea-49f6-b8bc-56d9a4a44ec1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.442420 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.442683 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca" pod "9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" (UID: "9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.444019 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4" pod "9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" (UID: "9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8"). InnerVolumeSpecName "kube-api-access-sdnx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.445616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.444324 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8" pod "3d868523-6bea-49f6-b8bc-56d9a4a44ec1" (UID: "3d868523-6bea-49f6-b8bc-56d9a4a44ec1"). InnerVolumeSpecName "kube-api-access-s5cs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.544480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.544448 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cs8\" (UniqueName: \"kubernetes.io/projected/3d868523-6bea-49f6-b8bc-56d9a4a44ec1-kube-api-access-s5cs8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.544480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.544474 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnx4\" (UniqueName: \"kubernetes.io/projected/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-kube-api-access-sdnx4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.544480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.544486 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.990155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.990099 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" event={"ID":"9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8","Type":"ContainerDied","Data":"7ba7ebf510891f4a5f725904197b73e22e0741b0c0e51e250d5e058c19f4d09e"} Nov 03 20:30:21.990790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.990768 5815 scope.go:122] "RemoveContainer" containerID="db8571b12f9f31abd4c8ef4201f92e72b617af24efd9ec11e5ec8367e0fd3789" Nov 03 20:30:21.997742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:21.993629 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl" Nov 03 20:30:22.088672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.076850 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" event={"ID":"3d868523-6bea-49f6-b8bc-56d9a4a44ec1","Type":"ContainerDied","Data":"13391665e2474da2497026d64a816152c677f45b1673e881cb63391302dd80f8"} Nov 03 20:30:22.088672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.077105 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2" Nov 03 20:30:22.092735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.092449 5815 scope.go:122] "RemoveContainer" containerID="61f4b03d92219ec39369262061a8e0390adfd54e63b3169d905a5962980b7fee" Nov 03 20:30:22.130420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.130390 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl"] Nov 03 20:30:22.140055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.140021 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895630-7g6dl"] Nov 03 20:30:22.155195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.155159 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2"] Nov 03 20:30:22.159161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.159129 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3211a0-29895630-hcvw2"] Nov 03 20:30:22.339171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.339144 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:22.377244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.377213 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca\") pod \"fdf3353e-bc17-4267-b45b-79f0b4d13352\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " Nov 03 20:30:22.377434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.377421 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9\" (UniqueName: \"kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9\") pod \"fdf3353e-bc17-4267-b45b-79f0b4d13352\" (UID: \"fdf3353e-bc17-4267-b45b-79f0b4d13352\") " Nov 03 20:30:22.378816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.378783 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca" pod "fdf3353e-bc17-4267-b45b-79f0b4d13352" (UID: "fdf3353e-bc17-4267-b45b-79f0b4d13352"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:22.380319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.380294 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9" pod "fdf3353e-bc17-4267-b45b-79f0b4d13352" (UID: "fdf3353e-bc17-4267-b45b-79f0b4d13352"). InnerVolumeSpecName "kube-api-access-6krv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:22.480179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.480147 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf3353e-bc17-4267-b45b-79f0b4d13352-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.480179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.480182 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krv9\" (UniqueName: \"kubernetes.io/projected/fdf3353e-bc17-4267-b45b-79f0b4d13352-kube-api-access-6krv9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.590863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.590781 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" path="/var/lib/kubelet/pods/07ed922e-05c1-4bf1-a826-593f11758f04/volumes" Nov 03 20:30:22.592020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.591994 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" path="/var/lib/kubelet/pods/3d868523-6bea-49f6-b8bc-56d9a4a44ec1/volumes" Nov 03 20:30:22.592627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.592608 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" path="/var/lib/kubelet/pods/3fa298ac-ad45-4159-814e-5f339b7e9c01/volumes" Nov 03 20:30:22.593685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.593664 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac25daf-4025-40c5-b675-368805684a43" path="/var/lib/kubelet/pods/4ac25daf-4025-40c5-b675-368805684a43/volumes" Nov 03 20:30:22.594302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.594283 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" path="/var/lib/kubelet/pods/9a16b908-3cf0-4e38-9cf4-8c339c642887/volumes" Nov 03 20:30:22.594909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.594889 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" path="/var/lib/kubelet/pods/9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8/volumes" Nov 03 20:30:22.595900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:22.595876 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" path="/var/lib/kubelet/pods/c442837a-8c84-40ad-b101-e8e3d89087ce/volumes" Nov 03 20:30:23.098464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:23.098437 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" Nov 03 20:30:23.098996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:23.098467 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78" event={"ID":"fdf3353e-bc17-4267-b45b-79f0b4d13352","Type":"ContainerDied","Data":"0b4c371f0d8921f18e9d9c3d0b180707f704487ebf91f105ef314c634202caa2"} Nov 03 20:30:23.098996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:23.098541 5815 scope.go:122] "RemoveContainer" containerID="ddd65aa6d32f8744393ba82cd144b4939b5b977f0b36d377d821583391a4cbff" Nov 03 20:30:23.119751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:23.119723 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78"] Nov 03 20:30:23.124704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:23.124683 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895630-6sr78"] Nov 03 20:30:24.588720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:24.588690 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" path="/var/lib/kubelet/pods/fdf3353e-bc17-4267-b45b-79f0b4d13352/volumes" Nov 03 20:30:26.032238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:26.032205 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v"] Nov 03 20:30:26.037118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:26.037093 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d53621-9272v"] Nov 03 20:30:26.590357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:26.590324 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecb1f63-8989-448f-a648-32d0e677d708" path="/var/lib/kubelet/pods/cecb1f63-8989-448f-a648-32d0e677d708/volumes" Nov 03 20:30:27.043576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:27.040104 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2"] Nov 03 20:30:27.047486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:27.047406 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2"] Nov 03 20:30:27.052373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:27.052345 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddb34247-wj4g2"] Nov 03 20:30:27.056926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:27.056905 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:30:27.061086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:27.061063 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82505810-jrht2"] Nov 03 20:30:29.064549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.063267 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3114265f-061d-4a4d-b6c6-e7b1b1e215d1" path="/var/lib/kubelet/pods/3114265f-061d-4a4d-b6c6-e7b1b1e215d1/volumes" Nov 03 20:30:29.081592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.075038 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1cff36-3f99-4fbf-83e0-3802126ab5df" path="/var/lib/kubelet/pods/dc1cff36-3f99-4fbf-83e0-3802126ab5df/volumes" Nov 03 20:30:29.242533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.233651 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:30:29.242842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.242803 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:29.470946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.470908 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:29.685532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.683667 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:29.685532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.684939 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/ready\": context deadline exceeded" Nov 03 20:30:29.708793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:29.708754 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:30.051579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:30.050574 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": context deadline exceeded" Nov 03 20:30:30.067600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:30.066763 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:30.146427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:30.146383 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:37.843288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:37.843257 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4"] Nov 03 20:30:42.067943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:42.067912 5815 scope.go:122] "RemoveContainer" containerID="abba51966699ec974c5afc477c396e8052d2b0be97a2d2e2ea7a9b44fbde94d2" Nov 03 20:30:42.082836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:42.082814 5815 scope.go:122] "RemoveContainer" containerID="2aecf42386a1f3d6c9e3bbd32f8a29648b4dce5aa50889cdc9527b26a366b4fb" Nov 03 20:30:42.094106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:42.094082 5815 scope.go:122] "RemoveContainer" containerID="13d64391b914d6e752eecb6240bc37659139ca59be9f020543dce79ff529f36b" Nov 03 20:30:42.105899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:42.105874 5815 scope.go:122] "RemoveContainer" containerID="216d983378d96dac70b963ac4d103d740fbbc0cccbe101836a25cd4e68510130" Nov 03 20:30:43.046551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:43.045595 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf"] Nov 03 20:30:43.062533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:43.059257 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38de8d6-jgdgf"] Nov 03 20:30:43.938146 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:43.938099 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:30:44.026526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.026475 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2"] Nov 03 20:30:44.032001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.031972 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh"] Nov 03 20:30:44.036211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.036187 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5fa8100-225t2"] Nov 03 20:30:44.040417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.040397 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b707df2c-r2jjh"] Nov 03 20:30:44.590852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.590817 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d546894-0603-4437-acfe-56632291e361" path="/var/lib/kubelet/pods/9d546894-0603-4437-acfe-56632291e361/volumes" Nov 03 20:30:44.591500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.591479 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7edac27-bd6a-4c3d-8e40-9d7230628363" path="/var/lib/kubelet/pods/a7edac27-bd6a-4c3d-8e40-9d7230628363/volumes" Nov 03 20:30:44.592136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:44.592115 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec51a1f-f96e-4436-8e9d-b3471beabd32" path="/var/lib/kubelet/pods/bec51a1f-f96e-4436-8e9d-b3471beabd32/volumes" Nov 03 20:30:48.032430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:48.032394 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd"] Nov 03 20:30:48.036991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:48.036961 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be838b-jzcnd"] Nov 03 20:30:48.589288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:48.589254 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2c7c60-8564-4bef-be38-3dd6c7ba03a4" path="/var/lib/kubelet/pods/3d2c7c60-8564-4bef-be38-3dd6c7ba03a4/volumes" Nov 03 20:30:52.873731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:52.873687 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" containerID="cri-o://639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4" gracePeriod=45 Nov 03 20:30:53.201052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.201026 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:30:53.280589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.280555 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs\") pod \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " Nov 03 20:30:53.280787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.280641 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d\" (UniqueName: \"kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d\") pod \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " Nov 03 20:30:53.281262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.281195 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache\") pod \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " Nov 03 20:30:53.281658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.281490 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache" pod "d88aa3a5-5e38-4cba-9cda-80134f0398a3" (UID: "d88aa3a5-5e38-4cba-9cda-80134f0398a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:53.281776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.281655 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home\") pod \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " Nov 03 20:30:53.281905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.281881 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home" pod "d88aa3a5-5e38-4cba-9cda-80134f0398a3" (UID: "d88aa3a5-5e38-4cba-9cda-80134f0398a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:53.282010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.281993 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir\") pod \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " Nov 03 20:30:53.282232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.282206 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir" pod "d88aa3a5-5e38-4cba-9cda-80134f0398a3" (UID: "d88aa3a5-5e38-4cba-9cda-80134f0398a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:53.282360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.282342 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm\") pod \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\" (UID: \"d88aa3a5-5e38-4cba-9cda-80134f0398a3\") " Nov 03 20:30:53.283752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.283726 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:53.283850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.283756 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:53.283850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.283772 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:53.284003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.283980 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs" pod "d88aa3a5-5e38-4cba-9cda-80134f0398a3" (UID: "d88aa3a5-5e38-4cba-9cda-80134f0398a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:53.285626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.284934 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d" pod "d88aa3a5-5e38-4cba-9cda-80134f0398a3" (UID: "d88aa3a5-5e38-4cba-9cda-80134f0398a3"). InnerVolumeSpecName "kube-api-access-gxj9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:53.312740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.312712 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm" pod "d88aa3a5-5e38-4cba-9cda-80134f0398a3" (UID: "d88aa3a5-5e38-4cba-9cda-80134f0398a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:53.421692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.404994 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88aa3a5-5e38-4cba-9cda-80134f0398a3-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:53.421692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.421583 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxj9d\" (UniqueName: \"kubernetes.io/projected/d88aa3a5-5e38-4cba-9cda-80134f0398a3-kube-api-access-gxj9d\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:53.421692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.421611 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88aa3a5-5e38-4cba-9cda-80134f0398a3-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:53.974636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.974604 5815 generic.go:356] "Generic (PLEG): container finished" podID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerID="639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4" exitCode=0 Nov 03 20:30:53.975075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.974650 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" Nov 03 20:30:53.975075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.974698 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" event={"ID":"d88aa3a5-5e38-4cba-9cda-80134f0398a3","Type":"ContainerDied","Data":"639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4"} Nov 03 20:30:53.975075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.974743 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4" event={"ID":"d88aa3a5-5e38-4cba-9cda-80134f0398a3","Type":"ContainerDied","Data":"be82cba0b1c661c8118ca665b0f2c45b8c3ab124ce982e2427b5099a758ed0ad"} Nov 03 20:30:53.975075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.974764 5815 scope.go:122] "RemoveContainer" containerID="639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4" Nov 03 20:30:53.999407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.999378 5815 scope.go:122] "RemoveContainer" containerID="639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4" Nov 03 20:30:53.999754 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:30:53.999734 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4\": container with ID starting with 639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4 not found: ID does not exist" containerID="639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4" Nov 03 20:30:53.999845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:53.999765 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4"} err="failed to get container status \"639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4\": rpc error: code = NotFound desc = could not find container \"639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4\": container with ID starting with 639d74d47d6640e102b27cc70d7d92bfc6fe2bded2dfb2a1e4999ca06842dff4 not found: ID does not exist" Nov 03 20:30:54.006260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:54.006237 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4"] Nov 03 20:30:54.013173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:54.013148 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-3211a0/praxis-route-3211a0-kserve-785759cc4f-r27c4"] Nov 03 20:30:54.590098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:30:54.590063 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" path="/var/lib/kubelet/pods/d88aa3a5-5e38-4cba-9cda-80134f0398a3/volumes" Nov 03 20:31:05.208187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:05.207052 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:31:08.709356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.709325 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp"] Nov 03 20:31:08.710188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710167 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710193 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710207 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:08.710237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710215 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:08.710237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710226 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:08.710237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710234 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710246 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710253 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710266 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710273 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710286 5815 cpu_manager.go:414] "containerMap: removing container" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710293 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710304 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710312 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710323 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710329 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710345 5815 cpu_manager.go:414] "containerMap: removing container" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710351 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710368 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710375 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710394 5815 cpu_manager.go:414] "containerMap: removing container" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710401 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710413 5815 cpu_manager.go:414] "containerMap: removing container" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710420 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710431 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710437 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710446 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710453 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710463 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710471 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710497 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710523 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710540 5815 cpu_manager.go:414] "containerMap: removing container" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710547 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710567 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710574 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710584 5815 cpu_manager.go:414] "containerMap: removing container" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710590 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710609 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710616 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710631 5815 cpu_manager.go:414] "containerMap: removing container" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710637 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710646 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710652 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710665 5815 cpu_manager.go:414] "containerMap: removing container" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710672 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710693 5815 cpu_manager.go:414] "containerMap: removing container" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710698 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710706 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710711 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710854 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710863 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710869 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710875 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710880 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710886 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710891 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710898 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710904 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710909 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710914 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710922 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710928 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710934 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710939 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710947 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:08.710934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710953 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.712334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710960 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ed922e-05c1-4bf1-a826-593f11758f04" containerName="cleanup" Nov 03 20:31:08.712334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710967 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:08.712334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710971 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:08.712334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710977 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:08.712334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710984 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88aa3a5-5e38-4cba-9cda-80134f0398a3" containerName="main" Nov 03 20:31:08.712334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.710991 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:08.714762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.714743 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.744221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.744193 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:31:08.748537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.744470 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:08.755050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.741028 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp"] Nov 03 20:31:08.790475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.790435 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtsbk\" (UniqueName: \"kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.791141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.791111 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.799535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.791380 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.903569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.902241 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.903569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.902445 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtsbk\" (UniqueName: \"kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.903569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.902765 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.903904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.903830 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.907239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.907186 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:08.917142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:08.917106 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtsbk\" (UniqueName: \"kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk\") pod \"maas-api-revoke-sub-75ca193e-h9gkp\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:09.029683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:09.029655 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7"] Nov 03 20:31:09.037037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:09.037009 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f86a8ac9-t7fk7"] Nov 03 20:31:09.087429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:09.087402 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:09.381230 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:09.381201 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982157e2_be88_458d_b66b_3caab90b7141.slice/crio-cbd562225385e9eb3fdd350145696e31f93c5da6aa9a9de1a131e4cdba7fcef0 WatchSource:0}: Error finding container cbd562225385e9eb3fdd350145696e31f93c5da6aa9a9de1a131e4cdba7fcef0: Status 404 returned error can't find the container with id cbd562225385e9eb3fdd350145696e31f93c5da6aa9a9de1a131e4cdba7fcef0 Nov 03 20:31:09.382200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:09.382175 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp"] Nov 03 20:31:10.029086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.028991 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6"] Nov 03 20:31:10.032015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.031991 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b"] Nov 03 20:31:10.038355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.038334 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d8c33cf-c8nb6"] Nov 03 20:31:10.044547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.044524 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277b3255-zbl8b"] Nov 03 20:31:10.303161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.303133 5815 generic.go:356] "Generic (PLEG): container finished" podID="982157e2-be88-458d-b66b-3caab90b7141" containerID="f55155f4b7aa8759762fd58d452fae84c48672add53d8770585e0cb97598c790" exitCode=0 Nov 03 20:31:10.303343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.303206 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" event={"ID":"982157e2-be88-458d-b66b-3caab90b7141","Type":"ContainerDied","Data":"f55155f4b7aa8759762fd58d452fae84c48672add53d8770585e0cb97598c790"} Nov 03 20:31:10.303343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.303254 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" event={"ID":"982157e2-be88-458d-b66b-3caab90b7141","Type":"ContainerStarted","Data":"cbd562225385e9eb3fdd350145696e31f93c5da6aa9a9de1a131e4cdba7fcef0"} Nov 03 20:31:10.589321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.589242 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26df59cc-659a-4c25-864c-7669e32e7593" path="/var/lib/kubelet/pods/26df59cc-659a-4c25-864c-7669e32e7593/volumes" Nov 03 20:31:10.589788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.589772 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951bf3c3-0618-41ed-a53d-485649fdc4f7" path="/var/lib/kubelet/pods/951bf3c3-0618-41ed-a53d-485649fdc4f7/volumes" Nov 03 20:31:10.590224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:10.590210 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d8e1d8-0361-4bd2-a93e-8fd079c3c470" path="/var/lib/kubelet/pods/c1d8e1d8-0361-4bd2-a93e-8fd079c3c470/volumes" Nov 03 20:31:11.026231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.026197 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq"] Nov 03 20:31:11.030543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.030497 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52586cb9-cjlfq"] Nov 03 20:31:11.532540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.518695 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:11.705763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.705264 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk\" (UniqueName: \"kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk\") pod \"982157e2-be88-458d-b66b-3caab90b7141\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " Nov 03 20:31:11.705763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.705466 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca\") pod \"982157e2-be88-458d-b66b-3caab90b7141\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " Nov 03 20:31:11.706783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.706397 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca\") pod \"982157e2-be88-458d-b66b-3caab90b7141\" (UID: \"982157e2-be88-458d-b66b-3caab90b7141\") " Nov 03 20:31:11.707465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.707436 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca" pod "982157e2-be88-458d-b66b-3caab90b7141" (UID: "982157e2-be88-458d-b66b-3caab90b7141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:11.708089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.708064 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/982157e2-be88-458d-b66b-3caab90b7141-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:11.711058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.711001 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk" pod "982157e2-be88-458d-b66b-3caab90b7141" (UID: "982157e2-be88-458d-b66b-3caab90b7141"). InnerVolumeSpecName "kube-api-access-qtsbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:11.711612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.711586 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca" pod "982157e2-be88-458d-b66b-3caab90b7141" (UID: "982157e2-be88-458d-b66b-3caab90b7141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:11.810219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.810187 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtsbk\" (UniqueName: \"kubernetes.io/projected/982157e2-be88-458d-b66b-3caab90b7141-kube-api-access-qtsbk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:11.810219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:11.810218 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/982157e2-be88-458d-b66b-3caab90b7141-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:12.331080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:12.329077 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" event={"ID":"982157e2-be88-458d-b66b-3caab90b7141","Type":"ContainerDied","Data":"cbd562225385e9eb3fdd350145696e31f93c5da6aa9a9de1a131e4cdba7fcef0"} Nov 03 20:31:12.331080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:12.329193 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd562225385e9eb3fdd350145696e31f93c5da6aa9a9de1a131e4cdba7fcef0" Nov 03 20:31:12.331080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:12.329262 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp" Nov 03 20:31:12.587988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:12.587901 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cec863f-dd73-40d1-8b89-3f6fb872a81f" path="/var/lib/kubelet/pods/8cec863f-dd73-40d1-8b89-3f6fb872a81f/volumes" Nov 03 20:31:13.924876 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:13.924829 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:31:14.918151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.918118 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs"] Nov 03 20:31:14.919001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.918979 5815 cpu_manager.go:414] "containerMap: removing container" podUID="982157e2-be88-458d-b66b-3caab90b7141" containerName="revoke-keys" Nov 03 20:31:14.919001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.918999 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="982157e2-be88-458d-b66b-3caab90b7141" containerName="revoke-keys" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919016 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919024 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919067 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919074 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919092 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919100 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919136 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919145 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919164 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:14.919191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919171 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:14.919885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919226 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:14.919885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.919235 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f13de40-0a6a-443c-a0ec-0c3c07f5e1c8" containerName="cleanup" Nov 03 20:31:14.921537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921495 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde1671e-7d58-4a8d-b2a0-e809c8fe66e6" containerName="cleanup" Nov 03 20:31:14.921537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921537 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac25daf-4025-40c5-b675-368805684a43" containerName="cleanup" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921548 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="982157e2-be88-458d-b66b-3caab90b7141" containerName="revoke-keys" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921565 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a16b908-3cf0-4e38-9cf4-8c339c642887" containerName="cleanup" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921582 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa298ac-ad45-4159-814e-5f339b7e9c01" containerName="cleanup" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921602 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98a3c1e-2c39-4eb2-9966-4d72a8b2192c" containerName="cleanup" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921610 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf3353e-bc17-4267-b45b-79f0b4d13352" containerName="cleanup" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921620 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d868523-6bea-49f6-b8bc-56d9a4a44ec1" containerName="cleanup" Nov 03 20:31:14.921717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:14.921631 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c442837a-8c84-40ad-b101-e8e3d89087ce" containerName="cleanup" Nov 03 20:31:15.012847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.012814 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.024805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.024649 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:15.028756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.028718 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:31:15.072290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.072256 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs"] Nov 03 20:31:15.096236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.096119 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb"] Nov 03 20:31:15.112303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.112265 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.112486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.112354 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftghd\" (UniqueName: \"kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.112486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.112419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.252540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.250923 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.252540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.251042 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftghd\" (UniqueName: \"kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.252540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.251129 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.252540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.252392 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.317708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.297964 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.317708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.312375 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftghd\" (UniqueName: \"kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd\") pod \"maas-api-revoke-sub-4e345ebc-bwdhs\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.378967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.377792 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:15.959108 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:15.959064 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367fcb0b_09d4_4aec_801a_33444de28026.slice/crio-5e1fb70d0cb701ee38f0eb38cdb954f43f14de5a1d57cfe08cd4c24339db8afc WatchSource:0}: Error finding container 5e1fb70d0cb701ee38f0eb38cdb954f43f14de5a1d57cfe08cd4c24339db8afc: Status 404 returned error can't find the container with id 5e1fb70d0cb701ee38f0eb38cdb954f43f14de5a1d57cfe08cd4c24339db8afc Nov 03 20:31:15.959746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:15.959578 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs"] Nov 03 20:31:16.561938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:16.561849 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" event={"ID":"367fcb0b-09d4-4aec-801a-33444de28026","Type":"ContainerStarted","Data":"3427088eddd691624fa192567be98558f92dffc624bc95d1920780b4ff3b1efc"} Nov 03 20:31:16.561938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:16.561907 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" event={"ID":"367fcb0b-09d4-4aec-801a-33444de28026","Type":"ContainerStarted","Data":"5e1fb70d0cb701ee38f0eb38cdb954f43f14de5a1d57cfe08cd4c24339db8afc"} Nov 03 20:31:16.581900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:16.581853 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" podStartSLOduration=2.581838752 podStartE2EDuration="2.581838752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:16.580179765 +0000 UTC m=+1786.110525439" watchObservedRunningTime="2026-11-03 20:31:16.581838752 +0000 UTC m=+1786.112184472" Nov 03 20:31:17.576896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:17.576861 5815 generic.go:356] "Generic (PLEG): container finished" podID="367fcb0b-09d4-4aec-801a-33444de28026" containerID="3427088eddd691624fa192567be98558f92dffc624bc95d1920780b4ff3b1efc" exitCode=0 Nov 03 20:31:17.577310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:17.576962 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" event={"ID":"367fcb0b-09d4-4aec-801a-33444de28026","Type":"ContainerDied","Data":"3427088eddd691624fa192567be98558f92dffc624bc95d1920780b4ff3b1efc"} Nov 03 20:31:18.785179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.784814 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:18.847612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.839196 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca\") pod \"367fcb0b-09d4-4aec-801a-33444de28026\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " Nov 03 20:31:18.847612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.839279 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd\" (UniqueName: \"kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd\") pod \"367fcb0b-09d4-4aec-801a-33444de28026\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " Nov 03 20:31:18.847612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.839358 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca\") pod \"367fcb0b-09d4-4aec-801a-33444de28026\" (UID: \"367fcb0b-09d4-4aec-801a-33444de28026\") " Nov 03 20:31:18.847612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.841436 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca" pod "367fcb0b-09d4-4aec-801a-33444de28026" (UID: "367fcb0b-09d4-4aec-801a-33444de28026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:18.852921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.852878 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca" pod "367fcb0b-09d4-4aec-801a-33444de28026" (UID: "367fcb0b-09d4-4aec-801a-33444de28026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:18.853693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.853668 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd" pod "367fcb0b-09d4-4aec-801a-33444de28026" (UID: "367fcb0b-09d4-4aec-801a-33444de28026"). InnerVolumeSpecName "kube-api-access-ftghd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:18.942707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.942675 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.942871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.942711 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftghd\" (UniqueName: \"kubernetes.io/projected/367fcb0b-09d4-4aec-801a-33444de28026-kube-api-access-ftghd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:18.942871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:18.942727 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/367fcb0b-09d4-4aec-801a-33444de28026-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:19.019974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.019942 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq"] Nov 03 20:31:19.020798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.020775 5815 cpu_manager.go:414] "containerMap: removing container" podUID="367fcb0b-09d4-4aec-801a-33444de28026" containerName="revoke-keys" Nov 03 20:31:19.020798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.020796 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367fcb0b-09d4-4aec-801a-33444de28026" containerName="revoke-keys" Nov 03 20:31:19.021034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.021019 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="367fcb0b-09d4-4aec-801a-33444de28026" containerName="revoke-keys" Nov 03 20:31:19.025038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.025014 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.034717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.034688 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq"] Nov 03 20:31:19.150274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.150201 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.150422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.150363 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2wz\" (UniqueName: \"kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.150546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.150492 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.263538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.255029 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.263538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.255185 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2wz\" (UniqueName: \"kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.263538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.255306 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.263538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.256231 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.276619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.271033 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.284802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.284769 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2wz\" (UniqueName: \"kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz\") pod \"maas-api-revoke-sub-a668a642-hhfcq\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.352094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.352059 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:19.603037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.603008 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" Nov 03 20:31:19.603663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.603035 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs" event={"ID":"367fcb0b-09d4-4aec-801a-33444de28026","Type":"ContainerDied","Data":"5e1fb70d0cb701ee38f0eb38cdb954f43f14de5a1d57cfe08cd4c24339db8afc"} Nov 03 20:31:19.606622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.606594 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e1fb70d0cb701ee38f0eb38cdb954f43f14de5a1d57cfe08cd4c24339db8afc" Nov 03 20:31:19.649298 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:19.649275 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee56785e_4c8a_40e7_8347_07c9413f245f.slice/crio-a7d531f3688c8349a483666163f26d0f25f85158fcd980517300808df657a6e4 WatchSource:0}: Error finding container a7d531f3688c8349a483666163f26d0f25f85158fcd980517300808df657a6e4: Status 404 returned error can't find the container with id a7d531f3688c8349a483666163f26d0f25f85158fcd980517300808df657a6e4 Nov 03 20:31:19.649776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:19.649750 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq"] Nov 03 20:31:20.096782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:20.096750 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerName="payload-pre-processing" containerID="cri-o://6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626" gracePeriod=25 Nov 03 20:31:20.614823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:20.614789 5815 generic.go:356] "Generic (PLEG): container finished" podID="ee56785e-4c8a-40e7-8347-07c9413f245f" containerID="3991d407927b63b89f66544d58d54db287f7786bd3407ffd2a83feee58bcf387" exitCode=0 Nov 03 20:31:20.615044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:20.614873 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" event={"ID":"ee56785e-4c8a-40e7-8347-07c9413f245f","Type":"ContainerDied","Data":"3991d407927b63b89f66544d58d54db287f7786bd3407ffd2a83feee58bcf387"} Nov 03 20:31:20.615044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:20.614934 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" event={"ID":"ee56785e-4c8a-40e7-8347-07c9413f245f","Type":"ContainerStarted","Data":"a7d531f3688c8349a483666163f26d0f25f85158fcd980517300808df657a6e4"} Nov 03 20:31:21.947884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:21.947856 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:22.126928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.126898 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz\" (UniqueName: \"kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz\") pod \"ee56785e-4c8a-40e7-8347-07c9413f245f\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " Nov 03 20:31:22.127123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.126981 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca\") pod \"ee56785e-4c8a-40e7-8347-07c9413f245f\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " Nov 03 20:31:22.127123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.127045 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca\") pod \"ee56785e-4c8a-40e7-8347-07c9413f245f\" (UID: \"ee56785e-4c8a-40e7-8347-07c9413f245f\") " Nov 03 20:31:22.127360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.127333 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca" pod "ee56785e-4c8a-40e7-8347-07c9413f245f" (UID: "ee56785e-4c8a-40e7-8347-07c9413f245f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:22.127847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.127738 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:22.129364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.129340 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca" pod "ee56785e-4c8a-40e7-8347-07c9413f245f" (UID: "ee56785e-4c8a-40e7-8347-07c9413f245f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:22.129566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.129546 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz" pod "ee56785e-4c8a-40e7-8347-07c9413f245f" (UID: "ee56785e-4c8a-40e7-8347-07c9413f245f"). InnerVolumeSpecName "kube-api-access-pr2wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:22.229034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.229006 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2wz\" (UniqueName: \"kubernetes.io/projected/ee56785e-4c8a-40e7-8347-07c9413f245f-kube-api-access-pr2wz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:22.229034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.229031 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee56785e-4c8a-40e7-8347-07c9413f245f-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:22.659690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.659596 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" event={"ID":"ee56785e-4c8a-40e7-8347-07c9413f245f","Type":"ContainerDied","Data":"a7d531f3688c8349a483666163f26d0f25f85158fcd980517300808df657a6e4"} Nov 03 20:31:22.659690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.659663 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7d531f3688c8349a483666163f26d0f25f85158fcd980517300808df657a6e4" Nov 03 20:31:22.659979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:22.659727 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq" Nov 03 20:31:23.030598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:23.030567 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c"] Nov 03 20:31:23.035298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:23.035273 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0e745a-h6k2c"] Nov 03 20:31:24.590991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:24.590959 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca9be54-6a5f-43ca-93a5-c6666c72719a" path="/var/lib/kubelet/pods/6ca9be54-6a5f-43ca-93a5-c6666c72719a/volumes" Nov 03 20:31:25.027618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:25.027583 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp"] Nov 03 20:31:25.032433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:25.032406 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69319350-x5wtp"] Nov 03 20:31:25.986450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:25.986399 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:31:26.590552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:26.590501 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b4f970-620c-47f4-9b37-a65f230f7835" path="/var/lib/kubelet/pods/e5b4f970-620c-47f4-9b37-a65f230f7835/volumes" Nov 03 20:31:31.777186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.777102 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx"] Nov 03 20:31:31.778180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.778152 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ee56785e-4c8a-40e7-8347-07c9413f245f" containerName="revoke-keys" Nov 03 20:31:31.778315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.778183 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56785e-4c8a-40e7-8347-07c9413f245f" containerName="revoke-keys" Nov 03 20:31:31.778375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.778332 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee56785e-4c8a-40e7-8347-07c9413f245f" containerName="revoke-keys" Nov 03 20:31:31.846565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.839196 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx"] Nov 03 20:31:31.846565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.839254 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c"] Nov 03 20:31:31.846565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.844222 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:31.853750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.852486 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:31:31.853750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.853172 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:31.877033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.877009 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:31.946659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:31.946617 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c"] Nov 03 20:31:32.216207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.008256 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v"] Nov 03 20:31:32.219070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.219043 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-7616a6"] Nov 03 20:31:32.241532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.116757 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.241532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.234545 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.241532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.234617 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjkx\" (UniqueName: \"kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.241532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.234686 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlx6w\" (UniqueName: \"kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.241532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.234749 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.241532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.234875 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.260737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.260710 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-7616a6"] Nov 03 20:31:32.261705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.261686 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:32.267137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.267115 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6p5pq\"" Nov 03 20:31:32.337051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337014 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.337257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337169 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7vq\" (UniqueName: \"kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq\") pod \"test-curl-7564-7616a6\" (UID: \"ef36aab0-d1e4-4084-b75c-30fb47b16f96\") " pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:32.337257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337234 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.337409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337349 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.337482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337416 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjkx\" (UniqueName: \"kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.337614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337478 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlx6w\" (UniqueName: \"kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.337686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337612 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.337994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.337970 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.338108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.338041 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.341450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.340964 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.341690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.341665 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.354277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.354247 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlx6w\" (UniqueName: \"kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w\") pod \"maas-api-revoke-sub-0312e9e3-jh68c\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.354661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.354640 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjkx\" (UniqueName: \"kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx\") pod \"maas-api-revoke-sub-aa7223b4-bjrkx\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.439240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.439205 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7vq\" (UniqueName: \"kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq\") pod \"test-curl-7564-7616a6\" (UID: \"ef36aab0-d1e4-4084-b75c-30fb47b16f96\") " pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:32.454178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.454146 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7vq\" (UniqueName: \"kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq\") pod \"test-curl-7564-7616a6\" (UID: \"ef36aab0-d1e4-4084-b75c-30fb47b16f96\") " pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:32.472919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.472657 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:32.575530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.574809 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:32.611645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.585026 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:32.879500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.879471 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx"] Nov 03 20:31:32.983424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.983393 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c"] Nov 03 20:31:32.990165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:32.990138 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-7616a6"] Nov 03 20:31:33.329496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:33.329410 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-7616a6" event={"ID":"ef36aab0-d1e4-4084-b75c-30fb47b16f96","Type":"ContainerStarted","Data":"9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0"} Nov 03 20:31:33.329749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:33.329727 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-7616a6" event={"ID":"ef36aab0-d1e4-4084-b75c-30fb47b16f96","Type":"ContainerStarted","Data":"988350049e0d041338cd2d834b90d200d9ed229ae77ff507cbe81b9f30161bf1"} Nov 03 20:31:33.332131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:33.332105 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" event={"ID":"cb942411-d605-46bb-8d86-06e12a8bd7b8","Type":"ContainerStarted","Data":"0714596dddac00a645e742c5f2f05237823a248711292e97dbb844d17c803d84"} Nov 03 20:31:33.334102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:33.334072 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" event={"ID":"16cb6436-8fdd-4c9a-a9c5-72f192c93c06","Type":"ContainerStarted","Data":"3d78ea7fa2d204a6b8b4d449bffe82f95dd85714ad466506bb7ff52f7fd32cbe"} Nov 03 20:31:33.350378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:33.350331 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7564-7616a6" podStartSLOduration=2.350317122 podStartE2EDuration="2.350317122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:33.344890456 +0000 UTC m=+1802.875236129" watchObservedRunningTime="2026-11-03 20:31:33.350317122 +0000 UTC m=+1802.880662795" Nov 03 20:31:34.048002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.047973 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-7616a6"] Nov 03 20:31:34.055185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.055156 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-7616a6"] Nov 03 20:31:34.232422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.232395 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk"] Nov 03 20:31:34.233201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.233183 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" containerName="test-curl-7564-7616a6" Nov 03 20:31:34.233261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.233205 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" containerName="test-curl-7564-7616a6" Nov 03 20:31:34.233396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.233386 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" containerName="test-curl-7564-7616a6" Nov 03 20:31:34.237169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.237147 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.246055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.246030 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk"] Nov 03 20:31:34.305295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.305219 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtzmq\" (UniqueName: \"kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.305295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.305259 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.305561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.305454 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.346889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.346859 5815 generic.go:356] "Generic (PLEG): container finished" podID="cb942411-d605-46bb-8d86-06e12a8bd7b8" containerID="c6939f296cefada88ddce1c7f8a0e00bafff52984130ca32ea5d4f3900707f9d" exitCode=0 Nov 03 20:31:34.347068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.346956 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" event={"ID":"cb942411-d605-46bb-8d86-06e12a8bd7b8","Type":"ContainerDied","Data":"c6939f296cefada88ddce1c7f8a0e00bafff52984130ca32ea5d4f3900707f9d"} Nov 03 20:31:34.348838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.348811 5815 generic.go:356] "Generic (PLEG): container finished" podID="16cb6436-8fdd-4c9a-a9c5-72f192c93c06" containerID="4259941579d545e3301ae95ca0618d81c651bd88000b8a4ede77b664de29dc14" exitCode=0 Nov 03 20:31:34.348966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.348884 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" event={"ID":"16cb6436-8fdd-4c9a-a9c5-72f192c93c06","Type":"ContainerDied","Data":"4259941579d545e3301ae95ca0618d81c651bd88000b8a4ede77b664de29dc14"} Nov 03 20:31:34.407239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.407202 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtzmq\" (UniqueName: \"kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.407437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.407258 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.407437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.407366 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.408022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.407997 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.410049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.410024 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.419899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.419877 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtzmq\" (UniqueName: \"kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq\") pod \"maas-api-revoke-sub-0b5fb11b-28plk\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.560980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.560902 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:34.884174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.884092 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk"] Nov 03 20:31:34.890709 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:34.890679 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fcc210_5af4_4b07_b2f7_4821c98866a5.slice/crio-e61042bf7b7a9286031d382a25683352af403dfe654e4365f18bc728ca6ecc3a WatchSource:0}: Error finding container e61042bf7b7a9286031d382a25683352af403dfe654e4365f18bc728ca6ecc3a: Status 404 returned error can't find the container with id e61042bf7b7a9286031d382a25683352af403dfe654e4365f18bc728ca6ecc3a Nov 03 20:31:34.955731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.955704 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:31:34.962342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.962320 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:31:34.969333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.969303 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:31:34.976239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:34.976218 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:31:35.364800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.364750 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" event={"ID":"49fcc210-5af4-4b07-b2f7-4821c98866a5","Type":"ContainerStarted","Data":"18b5a48140320e0f676b8d6773775704fcbce91a5e86c590cb63d3a61e14859d"} Nov 03 20:31:35.365281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.364812 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" event={"ID":"49fcc210-5af4-4b07-b2f7-4821c98866a5","Type":"ContainerStarted","Data":"e61042bf7b7a9286031d382a25683352af403dfe654e4365f18bc728ca6ecc3a"} Nov 03 20:31:35.365281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.364939 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7564-7616a6" podUID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" containerName="test-curl-7564-7616a6" containerID="cri-o://9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0" gracePeriod=2 Nov 03 20:31:35.389840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.389779 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" podStartSLOduration=1.389758998 podStartE2EDuration="1.389758998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:35.383840999 +0000 UTC m=+1804.914186676" watchObservedRunningTime="2026-11-03 20:31:35.389758998 +0000 UTC m=+1804.920104672" Nov 03 20:31:35.725247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.725218 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:35.818756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.818679 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:35.894698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.894667 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca\") pod \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " Nov 03 20:31:35.894928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.894775 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca\") pod \"cb942411-d605-46bb-8d86-06e12a8bd7b8\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " Nov 03 20:31:35.894928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.894851 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w\" (UniqueName: \"kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w\") pod \"cb942411-d605-46bb-8d86-06e12a8bd7b8\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " Nov 03 20:31:35.894928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.894905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx\" (UniqueName: \"kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx\") pod \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " Nov 03 20:31:35.895188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.895017 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca\") pod \"cb942411-d605-46bb-8d86-06e12a8bd7b8\" (UID: \"cb942411-d605-46bb-8d86-06e12a8bd7b8\") " Nov 03 20:31:35.895249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.895190 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca\") pod \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\" (UID: \"16cb6436-8fdd-4c9a-a9c5-72f192c93c06\") " Nov 03 20:31:35.896671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.896604 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca" pod "16cb6436-8fdd-4c9a-a9c5-72f192c93c06" (UID: "16cb6436-8fdd-4c9a-a9c5-72f192c93c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:35.897009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.896981 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca" pod "cb942411-d605-46bb-8d86-06e12a8bd7b8" (UID: "cb942411-d605-46bb-8d86-06e12a8bd7b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:35.899463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.899441 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w" pod "cb942411-d605-46bb-8d86-06e12a8bd7b8" (UID: "cb942411-d605-46bb-8d86-06e12a8bd7b8"). InnerVolumeSpecName "kube-api-access-zlx6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:35.899750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.899694 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca" pod "16cb6436-8fdd-4c9a-a9c5-72f192c93c06" (UID: "16cb6436-8fdd-4c9a-a9c5-72f192c93c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:35.900962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.900909 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca" pod "cb942411-d605-46bb-8d86-06e12a8bd7b8" (UID: "cb942411-d605-46bb-8d86-06e12a8bd7b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:35.901177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.901157 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx" pod "16cb6436-8fdd-4c9a-a9c5-72f192c93c06" (UID: "16cb6436-8fdd-4c9a-a9c5-72f192c93c06"). InnerVolumeSpecName "kube-api-access-lnjkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:35.986958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.986921 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:31:35.997438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.997402 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:35.997575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.997440 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:35.997575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.997455 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:35.997575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.997469 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlx6w\" (UniqueName: \"kubernetes.io/projected/cb942411-d605-46bb-8d86-06e12a8bd7b8-kube-api-access-zlx6w\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:35.997575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.997481 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnjkx\" (UniqueName: \"kubernetes.io/projected/16cb6436-8fdd-4c9a-a9c5-72f192c93c06-kube-api-access-lnjkx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:35.997575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:35.997495 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb942411-d605-46bb-8d86-06e12a8bd7b8-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:36.378240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.378210 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" Nov 03 20:31:36.379461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.378243 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx" event={"ID":"16cb6436-8fdd-4c9a-a9c5-72f192c93c06","Type":"ContainerDied","Data":"3d78ea7fa2d204a6b8b4d449bffe82f95dd85714ad466506bb7ff52f7fd32cbe"} Nov 03 20:31:36.379572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.379499 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d78ea7fa2d204a6b8b4d449bffe82f95dd85714ad466506bb7ff52f7fd32cbe" Nov 03 20:31:36.380783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.380752 5815 generic.go:356] "Generic (PLEG): container finished" podID="49fcc210-5af4-4b07-b2f7-4821c98866a5" containerID="18b5a48140320e0f676b8d6773775704fcbce91a5e86c590cb63d3a61e14859d" exitCode=0 Nov 03 20:31:36.380916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.380813 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" event={"ID":"49fcc210-5af4-4b07-b2f7-4821c98866a5","Type":"ContainerDied","Data":"18b5a48140320e0f676b8d6773775704fcbce91a5e86c590cb63d3a61e14859d"} Nov 03 20:31:36.391254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.391231 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" event={"ID":"cb942411-d605-46bb-8d86-06e12a8bd7b8","Type":"ContainerDied","Data":"0714596dddac00a645e742c5f2f05237823a248711292e97dbb844d17c803d84"} Nov 03 20:31:36.391395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.391257 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c" Nov 03 20:31:36.391395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.391278 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0714596dddac00a645e742c5f2f05237823a248711292e97dbb844d17c803d84" Nov 03 20:31:36.816477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.816444 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787"] Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.989609 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-a8e60d"] Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.991662 5815 cpu_manager.go:414] "containerMap: removing container" podUID="16cb6436-8fdd-4c9a-a9c5-72f192c93c06" containerName="revoke-keys" Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.991685 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cb6436-8fdd-4c9a-a9c5-72f192c93c06" containerName="revoke-keys" Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.991719 5815 cpu_manager.go:414] "containerMap: removing container" podUID="cb942411-d605-46bb-8d86-06e12a8bd7b8" containerName="revoke-keys" Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.991728 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb942411-d605-46bb-8d86-06e12a8bd7b8" containerName="revoke-keys" Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.992113 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cb6436-8fdd-4c9a-a9c5-72f192c93c06" containerName="revoke-keys" Nov 03 20:31:37.003545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:36.992131 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb942411-d605-46bb-8d86-06e12a8bd7b8" containerName="revoke-keys" Nov 03 20:31:37.041828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.041797 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:37.221614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.187672 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-a8e60d"] Nov 03 20:31:37.242247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.242190 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dln8r\" (UniqueName: \"kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r\") pod \"test-curl-7567-a8e60d\" (UID: \"3384da0d-8a11-434b-9a72-69ef21386b20\") " pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:37.343966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.343931 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dln8r\" (UniqueName: \"kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r\") pod \"test-curl-7567-a8e60d\" (UID: \"3384da0d-8a11-434b-9a72-69ef21386b20\") " pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:37.356946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.356918 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dln8r\" (UniqueName: \"kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r\") pod \"test-curl-7567-a8e60d\" (UID: \"3384da0d-8a11-434b-9a72-69ef21386b20\") " pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:37.511578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.510823 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:37.809941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.809741 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:37.813309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.813288 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:37.840833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.840801 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca\") pod \"49fcc210-5af4-4b07-b2f7-4821c98866a5\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " Nov 03 20:31:37.841006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.840874 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq\" (UniqueName: \"kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq\") pod \"ef36aab0-d1e4-4084-b75c-30fb47b16f96\" (UID: \"ef36aab0-d1e4-4084-b75c-30fb47b16f96\") " Nov 03 20:31:37.841006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.840960 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq\" (UniqueName: \"kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq\") pod \"49fcc210-5af4-4b07-b2f7-4821c98866a5\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " Nov 03 20:31:37.841145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.841084 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca\") pod \"49fcc210-5af4-4b07-b2f7-4821c98866a5\" (UID: \"49fcc210-5af4-4b07-b2f7-4821c98866a5\") " Nov 03 20:31:37.841222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.841193 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca" pod "49fcc210-5af4-4b07-b2f7-4821c98866a5" (UID: "49fcc210-5af4-4b07-b2f7-4821c98866a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:37.841923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.841895 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:37.844725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.844701 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca" pod "49fcc210-5af4-4b07-b2f7-4821c98866a5" (UID: "49fcc210-5af4-4b07-b2f7-4821c98866a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:37.845620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.845586 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq" pod "ef36aab0-d1e4-4084-b75c-30fb47b16f96" (UID: "ef36aab0-d1e4-4084-b75c-30fb47b16f96"). InnerVolumeSpecName "kube-api-access-mq7vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:37.847823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.847493 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq" pod "49fcc210-5af4-4b07-b2f7-4821c98866a5" (UID: "49fcc210-5af4-4b07-b2f7-4821c98866a5"). InnerVolumeSpecName "kube-api-access-vtzmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:37.943655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.943583 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7vq\" (UniqueName: \"kubernetes.io/projected/ef36aab0-d1e4-4084-b75c-30fb47b16f96-kube-api-access-mq7vq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:37.943655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.943618 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtzmq\" (UniqueName: \"kubernetes.io/projected/49fcc210-5af4-4b07-b2f7-4821c98866a5-kube-api-access-vtzmq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:37.943655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:37.943634 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49fcc210-5af4-4b07-b2f7-4821c98866a5-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:38.040963 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:38.040922 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3384da0d_8a11_434b_9a72_69ef21386b20.slice/crio-a3a17268d48a8f58cbac8b0c5ce81d9b320e6a17bed2228ac587dd062d89127b WatchSource:0}: Error finding container a3a17268d48a8f58cbac8b0c5ce81d9b320e6a17bed2228ac587dd062d89127b: Status 404 returned error can't find the container with id a3a17268d48a8f58cbac8b0c5ce81d9b320e6a17bed2228ac587dd062d89127b Nov 03 20:31:38.041287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.041259 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-a8e60d"] Nov 03 20:31:38.418780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.418747 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" Nov 03 20:31:38.418780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.418773 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk" event={"ID":"49fcc210-5af4-4b07-b2f7-4821c98866a5","Type":"ContainerDied","Data":"e61042bf7b7a9286031d382a25683352af403dfe654e4365f18bc728ca6ecc3a"} Nov 03 20:31:38.419053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.418823 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61042bf7b7a9286031d382a25683352af403dfe654e4365f18bc728ca6ecc3a" Nov 03 20:31:38.420432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.420406 5815 generic.go:356] "Generic (PLEG): container finished" podID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" containerID="9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0" exitCode=137 Nov 03 20:31:38.420614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.420438 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:38.420614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.420495 5815 scope.go:122] "RemoveContainer" containerID="9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0" Nov 03 20:31:38.426749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.426723 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-a8e60d" event={"ID":"3384da0d-8a11-434b-9a72-69ef21386b20","Type":"ContainerStarted","Data":"25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d"} Nov 03 20:31:38.426870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.426762 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-a8e60d" event={"ID":"3384da0d-8a11-434b-9a72-69ef21386b20","Type":"ContainerStarted","Data":"a3a17268d48a8f58cbac8b0c5ce81d9b320e6a17bed2228ac587dd062d89127b"} Nov 03 20:31:38.436966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.436950 5815 scope.go:122] "RemoveContainer" containerID="9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0" Nov 03 20:31:38.437308 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:38.437284 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0\": container with ID starting with 9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0 not found: ID does not exist" containerID="9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0" Nov 03 20:31:38.437370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.437319 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0"} err="failed to get container status \"9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0\": rpc error: code = NotFound desc = could not find container \"9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0\": container with ID starting with 9f2683a0ccc2f76adc4b8b01403aed6935888ae25add09b11bf48b27f3116dc0 not found: ID does not exist" Nov 03 20:31:38.449778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.449728 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7567-a8e60d" podStartSLOduration=2.449710742 podStartE2EDuration="2.449710742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:38.443956081 +0000 UTC m=+1807.974301755" watchObservedRunningTime="2026-11-03 20:31:38.449710742 +0000 UTC m=+1807.980056415" Nov 03 20:31:38.452993 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:38.452966 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-7616a6\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" pod="opendatahub/test-curl-7564-7616a6" Nov 03 20:31:38.617923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:38.615027 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef36aab0-d1e4-4084-b75c-30fb47b16f96" path="/var/lib/kubelet/pods/ef36aab0-d1e4-4084-b75c-30fb47b16f96/volumes" Nov 03 20:31:39.129609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.129576 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-a8e60d"] Nov 03 20:31:39.137163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.135631 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-a8e60d"] Nov 03 20:31:39.467595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.466605 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-829b9a"] Nov 03 20:31:39.467797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.467749 5815 cpu_manager.go:414] "containerMap: removing container" podUID="49fcc210-5af4-4b07-b2f7-4821c98866a5" containerName="revoke-keys" Nov 03 20:31:39.467797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.467773 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49fcc210-5af4-4b07-b2f7-4821c98866a5" containerName="revoke-keys" Nov 03 20:31:39.467916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.467849 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" containerName="test-curl-7567-a8e60d" Nov 03 20:31:39.467916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.467858 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" containerName="test-curl-7567-a8e60d" Nov 03 20:31:39.471545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.468052 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" containerName="test-curl-7567-a8e60d" Nov 03 20:31:39.471545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.468087 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="49fcc210-5af4-4b07-b2f7-4821c98866a5" containerName="revoke-keys" Nov 03 20:31:39.545560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.539776 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:39.595545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.590818 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz\") pod \"test-curl-7567-829b9a\" (UID: \"299f12e8-9799-4ae6-8837-12b77f6ec657\") " pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:39.728583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.727687 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz\") pod \"test-curl-7567-829b9a\" (UID: \"299f12e8-9799-4ae6-8837-12b77f6ec657\") " pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:39.832532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:39.831568 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-829b9a"] Nov 03 20:31:40.008114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.003917 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz\") pod \"test-curl-7567-829b9a\" (UID: \"299f12e8-9799-4ae6-8837-12b77f6ec657\") " pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:40.029952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.029916 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:40.318567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.318542 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:31:40.325136 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:40.325107 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299f12e8_9799_4ae6_8837_12b77f6ec657.slice/crio-16236385d4584a2416aa0dce4ad8045228fcaed932130ed363a8c63327d599f6 WatchSource:0}: Error finding container 16236385d4584a2416aa0dce4ad8045228fcaed932130ed363a8c63327d599f6: Status 404 returned error can't find the container with id 16236385d4584a2416aa0dce4ad8045228fcaed932130ed363a8c63327d599f6 Nov 03 20:31:40.326233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.326212 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-829b9a"] Nov 03 20:31:40.378823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.378796 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume\") pod \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " Nov 03 20:31:40.379091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.379068 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t\" (UniqueName: \"kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t\") pod \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\" (UID: \"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d\") " Nov 03 20:31:40.379599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.379575 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume" pod "c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" (UID: "c35f02ec-b38b-4ce7-9a6d-4165a64ed39d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:40.380427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.380401 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:40.382696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.382673 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t" pod "c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" (UID: "c35f02ec-b38b-4ce7-9a6d-4165a64ed39d"). InnerVolumeSpecName "kube-api-access-9p99t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:40.482965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.482928 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p99t\" (UniqueName: \"kubernetes.io/projected/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d-kube-api-access-9p99t\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:40.901600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.901464 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-829b9a" event={"ID":"299f12e8-9799-4ae6-8837-12b77f6ec657","Type":"ContainerStarted","Data":"e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9"} Nov 03 20:31:40.901600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.901558 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-829b9a" event={"ID":"299f12e8-9799-4ae6-8837-12b77f6ec657","Type":"ContainerStarted","Data":"16236385d4584a2416aa0dce4ad8045228fcaed932130ed363a8c63327d599f6"} Nov 03 20:31:40.903620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.903591 5815 generic.go:356] "Generic (PLEG): container finished" podID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerID="6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626" exitCode=0 Nov 03 20:31:40.903766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.903652 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" Nov 03 20:31:40.903766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.903697 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" event={"ID":"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d","Type":"ContainerDied","Data":"6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626"} Nov 03 20:31:40.903766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.903739 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb" event={"ID":"c35f02ec-b38b-4ce7-9a6d-4165a64ed39d","Type":"ContainerDied","Data":"6102df2be3a6200c44452ecc69741693684106b407c6904cbb4970177a0f4c6f"} Nov 03 20:31:40.903948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.903766 5815 scope.go:122] "RemoveContainer" containerID="6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626" Nov 03 20:31:40.904054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.904033 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7567-a8e60d" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" containerName="test-curl-7567-a8e60d" containerID="cri-o://25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d" gracePeriod=2 Nov 03 20:31:40.918391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.918364 5815 scope.go:122] "RemoveContainer" containerID="6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626" Nov 03 20:31:40.918742 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:40.918720 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626\": container with ID starting with 6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626 not found: ID does not exist" containerID="6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626" Nov 03 20:31:40.918845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.918755 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626"} err="failed to get container status \"6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626\": rpc error: code = NotFound desc = could not find container \"6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626\": container with ID starting with 6a69692227f25ebe894a02cfed1429da0b812020b2916bccbba0a9bf9d70f626 not found: ID does not exist" Nov 03 20:31:40.924796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.924751 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7567-829b9a" podStartSLOduration=1.9247380280000002 podStartE2EDuration="1.924738028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:40.921204691 +0000 UTC m=+1810.451550365" watchObservedRunningTime="2026-11-03 20:31:40.924738028 +0000 UTC m=+1810.455083741" Nov 03 20:31:40.939404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.939372 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb"] Nov 03 20:31:40.945162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:40.945138 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3211a0-75bb75df75-w6xkb"] Nov 03 20:31:41.594646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.594600 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-829b9a"] Nov 03 20:31:41.600662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.600629 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-829b9a"] Nov 03 20:31:41.758649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.758606 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-b0ef96"] Nov 03 20:31:41.759340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.759324 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerName="payload-pre-processing" Nov 03 20:31:41.759398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.759341 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerName="payload-pre-processing" Nov 03 20:31:41.759398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.759367 5815 cpu_manager.go:414] "containerMap: removing container" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" containerName="test-curl-7567-829b9a" Nov 03 20:31:41.759398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.759372 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" containerName="test-curl-7567-829b9a" Nov 03 20:31:41.759550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.759482 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" containerName="payload-pre-processing" Nov 03 20:31:41.759550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.759495 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" containerName="test-curl-7567-829b9a" Nov 03 20:31:41.764699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.764674 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:41.771606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.771580 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-b0ef96"] Nov 03 20:31:41.909287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:41.909191 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk74r\" (UniqueName: \"kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r\") pod \"test-curl-7567-b0ef96\" (UID: \"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88\") " pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:42.011881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.011803 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk74r\" (UniqueName: \"kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r\") pod \"test-curl-7567-b0ef96\" (UID: \"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88\") " pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:42.054068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.050972 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk74r\" (UniqueName: \"kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r\") pod \"test-curl-7567-b0ef96\" (UID: \"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88\") " pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:42.064583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.064545 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w"] Nov 03 20:31:42.089874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.084602 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.100811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.090860 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:42.101698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.101608 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:31:42.101698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.093868 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:42.113874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.113849 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9"] Nov 03 20:31:42.156342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.155827 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.156342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.155947 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbrkr\" (UniqueName: \"kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.156342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.156176 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.203808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.203730 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w"] Nov 03 20:31:42.204064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.204045 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9"] Nov 03 20:31:42.204315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.204293 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.258268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.258240 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.258380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.258286 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.258380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.258331 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tn5\" (UniqueName: \"kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.258558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.258379 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.258558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.258455 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbrkr\" (UniqueName: \"kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.258558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.258473 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.259599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.259574 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.265314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.265276 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.272473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.272451 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbrkr\" (UniqueName: \"kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr\") pod \"maas-api-revoke-sub-620191e8-fvs5w\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.342886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.342856 5815 scope.go:122] "RemoveContainer" containerID="4548dfe795e575b3e6d5db0ad6ec9caf851a88fc9703d03b84c83e9a4992c39a" Nov 03 20:31:42.355208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.355180 5815 scope.go:122] "RemoveContainer" containerID="2ee94bc5d3c07937c5029680301adb40db33aae3b391e9cf3f57e4801ff8ab97" Nov 03 20:31:42.359679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.359650 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.359796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.359730 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tn5\" (UniqueName: \"kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.359881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.359864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.360545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.360523 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.362131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.362109 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.366355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.366331 5815 scope.go:122] "RemoveContainer" containerID="2c4fd6bd873e2aa2640d1bc82fa7a41311355035adbdcbde00063ac4e7d89fa1" Nov 03 20:31:42.372576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.372547 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tn5\" (UniqueName: \"kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5\") pod \"maas-api-revoke-sub-ef6eaa03-9jbw9\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.377764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.377733 5815 scope.go:122] "RemoveContainer" containerID="49f3ccd181782709ce9491cac727e8a3bd9f3d7dd49fdb9403f602c0a6b599f9" Nov 03 20:31:42.388694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.388551 5815 scope.go:122] "RemoveContainer" containerID="845d4288ec8f2a7986b207527167e18f8c14017dae73c68b2dead9057de5c523" Nov 03 20:31:42.402173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.402155 5815 scope.go:122] "RemoveContainer" containerID="9558adf2b457c2ad9928994356158c5a401394682bcf47f4d3cd43dd176ea522" Nov 03 20:31:42.414296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.414275 5815 scope.go:122] "RemoveContainer" containerID="4d1d691304eacd6afb959c526764a487f57224ac2686e53a1af20b41c3c3fb8a" Nov 03 20:31:42.431161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.431141 5815 scope.go:122] "RemoveContainer" containerID="9a49aa17f891a944edcce0ed1f2f8342b0ea4bf1877ea884dc547650e80dc4f6" Nov 03 20:31:42.442602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.442585 5815 scope.go:122] "RemoveContainer" containerID="0d621fa9a9899f1ce0785c9005e642d6ec7b39f015b9038bb55667aa4ca81fe3" Nov 03 20:31:42.452804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.452785 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:42.453998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.453977 5815 scope.go:122] "RemoveContainer" containerID="98a381b0aa8e1e5793f43710a2bd533ef4365f303f0140548929a1155b6507f8" Nov 03 20:31:42.473650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.473622 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-b0ef96"] Nov 03 20:31:42.478792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.478769 5815 scope.go:122] "RemoveContainer" containerID="7b528e1b42fd15aaea4a783b22dea3fcc44aeae0ead8fe76c5bcf1e5abb9bd83" Nov 03 20:31:42.493046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.493026 5815 scope.go:122] "RemoveContainer" containerID="4965d3324937e84f7f3936cc6f89f2a3ac9e22e3568225ba0f23436ec1e13a1c" Nov 03 20:31:42.508171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.508138 5815 scope.go:122] "RemoveContainer" containerID="b70930781e02f5dd9a683b4188f0c42cee154840834199507ffc09b97127a7a1" Nov 03 20:31:42.532410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.532384 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:42.595852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.595804 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35f02ec-b38b-4ce7-9a6d-4165a64ed39d" path="/var/lib/kubelet/pods/c35f02ec-b38b-4ce7-9a6d-4165a64ed39d/volumes" Nov 03 20:31:42.777042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.777008 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w"] Nov 03 20:31:42.778604 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:42.778559 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670c716b_b0f2_463d_996f_3c67285e09a9.slice/crio-914d17f58efd71b294c158f64298c5bd95e712900cd8ec6813feeefdb4218fb2 WatchSource:0}: Error finding container 914d17f58efd71b294c158f64298c5bd95e712900cd8ec6813feeefdb4218fb2: Status 404 returned error can't find the container with id 914d17f58efd71b294c158f64298c5bd95e712900cd8ec6813feeefdb4218fb2 Nov 03 20:31:42.949531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.947340 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9"] Nov 03 20:31:42.951824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.951468 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" event={"ID":"670c716b-b0f2-463d-996f-3c67285e09a9","Type":"ContainerStarted","Data":"914d17f58efd71b294c158f64298c5bd95e712900cd8ec6813feeefdb4218fb2"} Nov 03 20:31:42.953861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.953829 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7567-829b9a" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" containerName="test-curl-7567-829b9a" containerID="cri-o://e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9" gracePeriod=2 Nov 03 20:31:42.954722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.954653 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-b0ef96" event={"ID":"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88","Type":"ContainerStarted","Data":"9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0"} Nov 03 20:31:42.954722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.954697 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-b0ef96" event={"ID":"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88","Type":"ContainerStarted","Data":"26139e720510d2189f3478ed6776654e59978f1d767d18dd89b4974fc268561e"} Nov 03 20:31:42.976667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:42.976589 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7567-b0ef96" podStartSLOduration=1.9765775319999999 podStartE2EDuration="1.976577532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:42.975790605 +0000 UTC m=+1812.506136277" watchObservedRunningTime="2026-11-03 20:31:42.976577532 +0000 UTC m=+1812.506923200" Nov 03 20:31:42.987999 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:42.987937 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc553e221_b343_41d6_b161_df2219a0f15d.slice/crio-df8a20fb9ba2187c56f88c62e316cffc03f097b32c7ead99f8bdfcc22ddea891 WatchSource:0}: Error finding container df8a20fb9ba2187c56f88c62e316cffc03f097b32c7ead99f8bdfcc22ddea891: Status 404 returned error can't find the container with id df8a20fb9ba2187c56f88c62e316cffc03f097b32c7ead99f8bdfcc22ddea891 Nov 03 20:31:43.229162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.229087 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:43.283024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.283002 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r\" (UniqueName: \"kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r\") pod \"3384da0d-8a11-434b-9a72-69ef21386b20\" (UID: \"3384da0d-8a11-434b-9a72-69ef21386b20\") " Nov 03 20:31:43.285332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.285307 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r" pod "3384da0d-8a11-434b-9a72-69ef21386b20" (UID: "3384da0d-8a11-434b-9a72-69ef21386b20"). InnerVolumeSpecName "kube-api-access-dln8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:43.385885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.385859 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dln8r\" (UniqueName: \"kubernetes.io/projected/3384da0d-8a11-434b-9a72-69ef21386b20-kube-api-access-dln8r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:43.831584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.829566 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-b0ef96"] Nov 03 20:31:43.836665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.835848 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-b0ef96"] Nov 03 20:31:43.864351 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:43.864300 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:31:43.966333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.966301 5815 generic.go:356] "Generic (PLEG): container finished" podID="3384da0d-8a11-434b-9a72-69ef21386b20" containerID="25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d" exitCode=137 Nov 03 20:31:43.966877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.966349 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:43.966877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.966408 5815 scope.go:122] "RemoveContainer" containerID="25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d" Nov 03 20:31:43.969051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.969024 5815 generic.go:356] "Generic (PLEG): container finished" podID="670c716b-b0f2-463d-996f-3c67285e09a9" containerID="e27a80245651b16c252a93e1a53cd3e5cdf0c427b7cf5b46600bff310f7de530" exitCode=0 Nov 03 20:31:43.969568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.969218 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" event={"ID":"670c716b-b0f2-463d-996f-3c67285e09a9","Type":"ContainerDied","Data":"e27a80245651b16c252a93e1a53cd3e5cdf0c427b7cf5b46600bff310f7de530"} Nov 03 20:31:43.979605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.979579 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" event={"ID":"c553e221-b343-41d6-b161-df2219a0f15d","Type":"ContainerStarted","Data":"f102dceb58688be1ad1e03cf990e2c3c5646f5b7ad16ba2975a806d95a1dc557"} Nov 03 20:31:43.979706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.979631 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" event={"ID":"c553e221-b343-41d6-b161-df2219a0f15d","Type":"ContainerStarted","Data":"df8a20fb9ba2187c56f88c62e316cffc03f097b32c7ead99f8bdfcc22ddea891"} Nov 03 20:31:43.983685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.983665 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-e20206"] Nov 03 20:31:43.984189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.984167 5815 scope.go:122] "RemoveContainer" containerID="25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d" Nov 03 20:31:43.984468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.984450 5815 cpu_manager.go:414] "containerMap: removing container" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" containerName="test-curl-7567-b0ef96" Nov 03 20:31:43.984609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.984469 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" containerName="test-curl-7567-b0ef96" Nov 03 20:31:43.984609 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:43.984497 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d\": container with ID starting with 25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d not found: ID does not exist" containerID="25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d" Nov 03 20:31:43.984609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.984579 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d"} err="failed to get container status \"25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d\": rpc error: code = NotFound desc = could not find container \"25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d\": container with ID starting with 25e02ab2c345365caffead2fe9a7b3ac2bd32bc97c9354c25b040c0d52aed70d not found: ID does not exist" Nov 03 20:31:43.984777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.984668 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" containerName="test-curl-7567-b0ef96" Nov 03 20:31:43.991435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.991414 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:43.996741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:43.996687 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-e20206"] Nov 03 20:31:44.025792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.025745 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" podStartSLOduration=2.025729064 podStartE2EDuration="2.025729064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:44.021239065 +0000 UTC m=+1813.551584739" watchObservedRunningTime="2026-11-03 20:31:44.025729064 +0000 UTC m=+1813.556074737" Nov 03 20:31:44.041568 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:44.039296 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-a8e60d\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:44.041568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.039781 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvcz\" (UniqueName: \"kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz\") pod \"test-curl-7567-e20206\" (UID: \"3295711e-1416-4bcd-9e12-2b44f1931dcf\") " pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:44.145140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.145103 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvcz\" (UniqueName: \"kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz\") pod \"test-curl-7567-e20206\" (UID: \"3295711e-1416-4bcd-9e12-2b44f1931dcf\") " pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:44.158058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.158031 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvcz\" (UniqueName: \"kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz\") pod \"test-curl-7567-e20206\" (UID: \"3295711e-1416-4bcd-9e12-2b44f1931dcf\") " pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:44.310686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.310650 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:44.512480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.512439 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft"] Nov 03 20:31:44.512907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.512874 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" containerID="cri-o://e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8" gracePeriod=30 Nov 03 20:31:44.532294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.532266 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8"] Nov 03 20:31:44.532773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.532728 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" containerID="cri-o://2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a" gracePeriod=30 Nov 03 20:31:44.540270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.540241 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5"] Nov 03 20:31:44.540461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.540438 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" containerID="cri-o://42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060" gracePeriod=30 Nov 03 20:31:44.556559 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:44.555671 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-a8e60d\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:44.558309 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:44.558282 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-a8e60d\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" pod="opendatahub/test-curl-7567-a8e60d" Nov 03 20:31:44.624671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.624630 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3384da0d-8a11-434b-9a72-69ef21386b20" path="/var/lib/kubelet/pods/3384da0d-8a11-434b-9a72-69ef21386b20/volumes" Nov 03 20:31:44.743695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.732200 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-e20206"] Nov 03 20:31:44.827535 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:44.822866 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3295711e_1416_4bcd_9e12_2b44f1931dcf.slice/crio-286eb64948bc43107e329f7ee4cff583e57a62d62e11564852b403a3ca485314 WatchSource:0}: Error finding container 286eb64948bc43107e329f7ee4cff583e57a62d62e11564852b403a3ca485314: Status 404 returned error can't find the container with id 286eb64948bc43107e329f7ee4cff583e57a62d62e11564852b403a3ca485314 Nov 03 20:31:44.951534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.934822 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9"] Nov 03 20:31:44.997322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:44.996145 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.172184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.160543 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.175346 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:45.175177 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299f12e8_9799_4ae6_8837_12b77f6ec657.slice/crio-conmon-e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299f12e8_9799_4ae6_8837_12b77f6ec657.slice/crio-e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:31:45.178088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.178057 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.178219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.178206 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p292l\" (UniqueName: \"kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.204728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.204701 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9"] Nov 03 20:31:45.211334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.211272 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-e20206" event={"ID":"3295711e-1416-4bcd-9e12-2b44f1931dcf","Type":"ContainerStarted","Data":"286eb64948bc43107e329f7ee4cff583e57a62d62e11564852b403a3ca485314"} Nov 03 20:31:45.214197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.214165 5815 generic.go:356] "Generic (PLEG): container finished" podID="c553e221-b343-41d6-b161-df2219a0f15d" containerID="f102dceb58688be1ad1e03cf990e2c3c5646f5b7ad16ba2975a806d95a1dc557" exitCode=0 Nov 03 20:31:45.214421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.214395 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" event={"ID":"c553e221-b343-41d6-b161-df2219a0f15d","Type":"ContainerDied","Data":"f102dceb58688be1ad1e03cf990e2c3c5646f5b7ad16ba2975a806d95a1dc557"} Nov 03 20:31:45.216289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.216262 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7567-b0ef96" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" containerName="test-curl-7567-b0ef96" containerID="cri-o://9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0" gracePeriod=2 Nov 03 20:31:45.281209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.281166 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.282111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.282086 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.283122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.283104 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p292l\" (UniqueName: \"kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.283862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.283038 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.285424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.284766 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.299745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.298123 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p292l\" (UniqueName: \"kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l\") pod \"maas-api-revoke-sub-51730991-44wd9\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.456432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.456402 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:45.459113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.459093 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:45.511563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.511497 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:45.589837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.589813 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz\") pod \"299f12e8-9799-4ae6-8837-12b77f6ec657\" (UID: \"299f12e8-9799-4ae6-8837-12b77f6ec657\") " Nov 03 20:31:45.589967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.589912 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca\") pod \"670c716b-b0f2-463d-996f-3c67285e09a9\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " Nov 03 20:31:45.589967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.589931 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr\" (UniqueName: \"kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr\") pod \"670c716b-b0f2-463d-996f-3c67285e09a9\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " Nov 03 20:31:45.590519 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.590297 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca" pod "670c716b-b0f2-463d-996f-3c67285e09a9" (UID: "670c716b-b0f2-463d-996f-3c67285e09a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:45.590519 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.590333 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca\") pod \"670c716b-b0f2-463d-996f-3c67285e09a9\" (UID: \"670c716b-b0f2-463d-996f-3c67285e09a9\") " Nov 03 20:31:45.591213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.591188 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.594139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.594104 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca" pod "670c716b-b0f2-463d-996f-3c67285e09a9" (UID: "670c716b-b0f2-463d-996f-3c67285e09a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:45.602134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.602098 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr" pod "670c716b-b0f2-463d-996f-3c67285e09a9" (UID: "670c716b-b0f2-463d-996f-3c67285e09a9"). InnerVolumeSpecName "kube-api-access-lbrkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:45.607075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.607037 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz" pod "299f12e8-9799-4ae6-8837-12b77f6ec657" (UID: "299f12e8-9799-4ae6-8837-12b77f6ec657"). InnerVolumeSpecName "kube-api-access-zztpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:45.692789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.692756 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbrkr\" (UniqueName: \"kubernetes.io/projected/670c716b-b0f2-463d-996f-3c67285e09a9-kube-api-access-lbrkr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.692789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.692787 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670c716b-b0f2-463d-996f-3c67285e09a9-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.692994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.692802 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/299f12e8-9799-4ae6-8837-12b77f6ec657-kube-api-access-zztpz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.812022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:45.811987 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9"] Nov 03 20:31:45.812397 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:45.812367 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb008bd53_0d87_4f40_890f_1ab3384913d8.slice/crio-6e32c6a5474ffff4bd54464ae0e76375d35fead9b584a392c0246853c820fed6 WatchSource:0}: Error finding container 6e32c6a5474ffff4bd54464ae0e76375d35fead9b584a392c0246853c820fed6: Status 404 returned error can't find the container with id 6e32c6a5474ffff4bd54464ae0e76375d35fead9b584a392c0246853c820fed6 Nov 03 20:31:46.226281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.226243 5815 generic.go:356] "Generic (PLEG): container finished" podID="299f12e8-9799-4ae6-8837-12b77f6ec657" containerID="e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9" exitCode=137 Nov 03 20:31:46.226808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.226296 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:46.226808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.226330 5815 scope.go:122] "RemoveContainer" containerID="e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9" Nov 03 20:31:46.228415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.228396 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" Nov 03 20:31:46.228415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.228406 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w" event={"ID":"670c716b-b0f2-463d-996f-3c67285e09a9","Type":"ContainerDied","Data":"914d17f58efd71b294c158f64298c5bd95e712900cd8ec6813feeefdb4218fb2"} Nov 03 20:31:46.228686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.228448 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914d17f58efd71b294c158f64298c5bd95e712900cd8ec6813feeefdb4218fb2" Nov 03 20:31:46.230768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.230741 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" event={"ID":"b008bd53-0d87-4f40-890f-1ab3384913d8","Type":"ContainerStarted","Data":"fa9d56580ba37dcf83e41a7d8b3fd78169dd01b0b58690ac4fb2c5e458f83073"} Nov 03 20:31:46.230891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.230795 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" event={"ID":"b008bd53-0d87-4f40-890f-1ab3384913d8","Type":"ContainerStarted","Data":"6e32c6a5474ffff4bd54464ae0e76375d35fead9b584a392c0246853c820fed6"} Nov 03 20:31:46.233223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.233200 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-e20206" event={"ID":"3295711e-1416-4bcd-9e12-2b44f1931dcf","Type":"ContainerStarted","Data":"b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d"} Nov 03 20:31:46.242302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.242285 5815 scope.go:122] "RemoveContainer" containerID="e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9" Nov 03 20:31:46.243737 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:46.243709 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9\": container with ID starting with e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9 not found: ID does not exist" containerID="e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9" Nov 03 20:31:46.243949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.243746 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9"} err="failed to get container status \"e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9\": rpc error: code = NotFound desc = could not find container \"e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9\": container with ID starting with e34473bc9768a7ac45801f5544bb3ee8c4768e343aff3d2cd822b443701316c9 not found: ID does not exist" Nov 03 20:31:46.251751 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:46.251724 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-829b9a\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:46.253106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.253067 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" podStartSLOduration=2.253054341 podStartE2EDuration="2.253054341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:46.249046525 +0000 UTC m=+1815.779392198" watchObservedRunningTime="2026-11-03 20:31:46.253054341 +0000 UTC m=+1815.783400017" Nov 03 20:31:46.274020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.273966 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7567-e20206" podStartSLOduration=3.2739487179999998 podStartE2EDuration="3.273948718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:46.270950249 +0000 UTC m=+1815.801295922" watchObservedRunningTime="2026-11-03 20:31:46.273948718 +0000 UTC m=+1815.804294388" Nov 03 20:31:46.432784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.432750 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:46.453617 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:46.453573 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-829b9a\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" pod="opendatahub/test-curl-7567-829b9a" Nov 03 20:31:46.508058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.508023 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca\") pod \"c553e221-b343-41d6-b161-df2219a0f15d\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " Nov 03 20:31:46.508336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.508309 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca\") pod \"c553e221-b343-41d6-b161-df2219a0f15d\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " Nov 03 20:31:46.508470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.508412 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5\" (UniqueName: \"kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5\") pod \"c553e221-b343-41d6-b161-df2219a0f15d\" (UID: \"c553e221-b343-41d6-b161-df2219a0f15d\") " Nov 03 20:31:46.508776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.508743 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca" pod "c553e221-b343-41d6-b161-df2219a0f15d" (UID: "c553e221-b343-41d6-b161-df2219a0f15d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:46.509753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.509724 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:46.511810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.511761 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5" pod "c553e221-b343-41d6-b161-df2219a0f15d" (UID: "c553e221-b343-41d6-b161-df2219a0f15d"). InnerVolumeSpecName "kube-api-access-b8tn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:46.511941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.511913 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca" pod "c553e221-b343-41d6-b161-df2219a0f15d" (UID: "c553e221-b343-41d6-b161-df2219a0f15d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:46.590225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.590188 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299f12e8-9799-4ae6-8837-12b77f6ec657" path="/var/lib/kubelet/pods/299f12e8-9799-4ae6-8837-12b77f6ec657/volumes" Nov 03 20:31:46.611693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.611666 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c553e221-b343-41d6-b161-df2219a0f15d-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:46.611693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:46.611691 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8tn5\" (UniqueName: \"kubernetes.io/projected/c553e221-b343-41d6-b161-df2219a0f15d-kube-api-access-b8tn5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.039882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.039824 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-e20206"] Nov 03 20:31:47.046081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.046052 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-e20206"] Nov 03 20:31:47.199076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.199045 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-3041f7"] Nov 03 20:31:47.200216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.200188 5815 cpu_manager.go:414] "containerMap: removing container" podUID="670c716b-b0f2-463d-996f-3c67285e09a9" containerName="revoke-keys" Nov 03 20:31:47.200397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.200382 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670c716b-b0f2-463d-996f-3c67285e09a9" containerName="revoke-keys" Nov 03 20:31:47.200540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.200527 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c553e221-b343-41d6-b161-df2219a0f15d" containerName="revoke-keys" Nov 03 20:31:47.200637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.200626 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c553e221-b343-41d6-b161-df2219a0f15d" containerName="revoke-keys" Nov 03 20:31:47.200760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.200748 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" containerName="test-curl-7567-e20206" Nov 03 20:31:47.200836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.200825 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" containerName="test-curl-7567-e20206" Nov 03 20:31:47.201164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.201146 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" containerName="test-curl-7567-e20206" Nov 03 20:31:47.238146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.238107 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="670c716b-b0f2-463d-996f-3c67285e09a9" containerName="revoke-keys" Nov 03 20:31:47.239532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.239495 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c553e221-b343-41d6-b161-df2219a0f15d" containerName="revoke-keys" Nov 03 20:31:47.246418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.246399 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-3041f7"] Nov 03 20:31:47.246670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.246646 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:47.250097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.250076 5815 generic.go:356] "Generic (PLEG): container finished" podID="b008bd53-0d87-4f40-890f-1ab3384913d8" containerID="fa9d56580ba37dcf83e41a7d8b3fd78169dd01b0b58690ac4fb2c5e458f83073" exitCode=0 Nov 03 20:31:47.250275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.250207 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" event={"ID":"b008bd53-0d87-4f40-890f-1ab3384913d8","Type":"ContainerDied","Data":"fa9d56580ba37dcf83e41a7d8b3fd78169dd01b0b58690ac4fb2c5e458f83073"} Nov 03 20:31:47.253729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.253534 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" Nov 03 20:31:47.253729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.253627 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9" event={"ID":"c553e221-b343-41d6-b161-df2219a0f15d","Type":"ContainerDied","Data":"df8a20fb9ba2187c56f88c62e316cffc03f097b32c7ead99f8bdfcc22ddea891"} Nov 03 20:31:47.253729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.253661 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df8a20fb9ba2187c56f88c62e316cffc03f097b32c7ead99f8bdfcc22ddea891" Nov 03 20:31:47.286015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.285964 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" containerID="cri-o://01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186" gracePeriod=45 Nov 03 20:31:47.328693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.328661 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpbq\" (UniqueName: \"kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq\") pod \"test-curl-7567-3041f7\" (UID: \"4c2fd8be-ad57-4cd5-b079-adfd0141054b\") " pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:47.431137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.431096 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpbq\" (UniqueName: \"kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq\") pod \"test-curl-7567-3041f7\" (UID: \"4c2fd8be-ad57-4cd5-b079-adfd0141054b\") " pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:47.444260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.444211 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpbq\" (UniqueName: \"kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq\") pod \"test-curl-7567-3041f7\" (UID: \"4c2fd8be-ad57-4cd5-b079-adfd0141054b\") " pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:47.532049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.531816 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:47.654587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.651394 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r\" (UniqueName: \"kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r\") pod \"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88\" (UID: \"25c9dea9-da8b-4ad8-a06f-0df2b14f2e88\") " Nov 03 20:31:47.681996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.679582 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r" pod "25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" (UID: "25c9dea9-da8b-4ad8-a06f-0df2b14f2e88"). InnerVolumeSpecName "kube-api-access-vk74r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:47.689783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.689457 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:47.722765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.722733 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:31:47.752950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.752922 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c\" (UniqueName: \"kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c\") pod \"2592b7c8-8386-4df0-8509-caf805154d4c\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " Nov 03 20:31:47.753092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.752972 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir\") pod \"2592b7c8-8386-4df0-8509-caf805154d4c\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " Nov 03 20:31:47.753168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.753148 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm\") pod \"2592b7c8-8386-4df0-8509-caf805154d4c\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " Nov 03 20:31:47.753774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.753236 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache\") pod \"2592b7c8-8386-4df0-8509-caf805154d4c\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " Nov 03 20:31:47.753774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.753332 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs\") pod \"2592b7c8-8386-4df0-8509-caf805154d4c\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " Nov 03 20:31:47.753774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.753467 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home\") pod \"2592b7c8-8386-4df0-8509-caf805154d4c\" (UID: \"2592b7c8-8386-4df0-8509-caf805154d4c\") " Nov 03 20:31:47.753774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.753657 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache" pod "2592b7c8-8386-4df0-8509-caf805154d4c" (UID: "2592b7c8-8386-4df0-8509-caf805154d4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:47.754059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.753918 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir" pod "2592b7c8-8386-4df0-8509-caf805154d4c" (UID: "2592b7c8-8386-4df0-8509-caf805154d4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:47.755012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.754755 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.755012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.754783 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk74r\" (UniqueName: \"kubernetes.io/projected/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88-kube-api-access-vk74r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.755012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.754810 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.755320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.755288 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home" pod "2592b7c8-8386-4df0-8509-caf805154d4c" (UID: "2592b7c8-8386-4df0-8509-caf805154d4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:47.756988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.756963 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm" pod "2592b7c8-8386-4df0-8509-caf805154d4c" (UID: "2592b7c8-8386-4df0-8509-caf805154d4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:47.757379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.757343 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c" pod "2592b7c8-8386-4df0-8509-caf805154d4c" (UID: "2592b7c8-8386-4df0-8509-caf805154d4c"). InnerVolumeSpecName "kube-api-access-tzw5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:47.758440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.758419 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs" pod "2592b7c8-8386-4df0-8509-caf805154d4c" (UID: "2592b7c8-8386-4df0-8509-caf805154d4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:47.856465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.856433 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.856636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.856470 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzw5c\" (UniqueName: \"kubernetes.io/projected/2592b7c8-8386-4df0-8509-caf805154d4c-kube-api-access-tzw5c\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.856636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.856485 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2592b7c8-8386-4df0-8509-caf805154d4c-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.856636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.856500 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2592b7c8-8386-4df0-8509-caf805154d4c-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:47.920592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.920490 5815 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:31:47.920592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:47.920569 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:31:48.021789 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:48.021758 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2fd8be_ad57_4cd5_b079_adfd0141054b.slice/crio-89a27762ebc23acdb3a739f6b8a0dfd0f90fc6cac66fed0366b9b9423e102a18 WatchSource:0}: Error finding container 89a27762ebc23acdb3a739f6b8a0dfd0f90fc6cac66fed0366b9b9423e102a18: Status 404 returned error can't find the container with id 89a27762ebc23acdb3a739f6b8a0dfd0f90fc6cac66fed0366b9b9423e102a18 Nov 03 20:31:48.022145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.022124 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-3041f7"] Nov 03 20:31:48.262909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.262881 5815 generic.go:356] "Generic (PLEG): container finished" podID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" containerID="9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0" exitCode=137 Nov 03 20:31:48.263417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.262933 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:48.263417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.262988 5815 scope.go:122] "RemoveContainer" containerID="9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0" Nov 03 20:31:48.264968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.264940 5815 generic.go:356] "Generic (PLEG): container finished" podID="2592b7c8-8386-4df0-8509-caf805154d4c" containerID="01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186" exitCode=0 Nov 03 20:31:48.265094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.264986 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" Nov 03 20:31:48.265094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.265018 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" event={"ID":"2592b7c8-8386-4df0-8509-caf805154d4c","Type":"ContainerDied","Data":"01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186"} Nov 03 20:31:48.265094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.265063 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v" event={"ID":"2592b7c8-8386-4df0-8509-caf805154d4c","Type":"ContainerDied","Data":"436ecd781124684cf8cd42a3fa861f9cd762a20268e9a0175e0ba915e4c63e8d"} Nov 03 20:31:48.267555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.267524 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7567-e20206" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" containerName="test-curl-7567-e20206" containerID="cri-o://b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d" gracePeriod=2 Nov 03 20:31:48.268222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.268195 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-3041f7" event={"ID":"4c2fd8be-ad57-4cd5-b079-adfd0141054b","Type":"ContainerStarted","Data":"c691e3f0c014f6f8235f4c5d15e78c188900479a4dbc50f03253e4296c5f26db"} Nov 03 20:31:48.268326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.268235 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-3041f7" event={"ID":"4c2fd8be-ad57-4cd5-b079-adfd0141054b","Type":"ContainerStarted","Data":"89a27762ebc23acdb3a739f6b8a0dfd0f90fc6cac66fed0366b9b9423e102a18"} Nov 03 20:31:48.272703 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.271476 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-e20206\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:48.278547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.278516 5815 scope.go:122] "RemoveContainer" containerID="9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0" Nov 03 20:31:48.278831 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.278809 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0\": container with ID starting with 9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0 not found: ID does not exist" containerID="9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0" Nov 03 20:31:48.278911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.278842 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0"} err="failed to get container status \"9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0\": rpc error: code = NotFound desc = could not find container \"9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0\": container with ID starting with 9a4dbd232cb01c33b0486c44df906465ac8131461a8ede5fe8fc8ba858c47db0 not found: ID does not exist" Nov 03 20:31:48.278911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.278865 5815 scope.go:122] "RemoveContainer" containerID="01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186" Nov 03 20:31:48.292776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.292728 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7567-3041f7" podStartSLOduration=1.292712539 podStartE2EDuration="1.292712539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:48.286783401 +0000 UTC m=+1817.817129074" watchObservedRunningTime="2026-11-03 20:31:48.292712539 +0000 UTC m=+1817.823058213" Nov 03 20:31:48.293137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.293043 5815 scope.go:122] "RemoveContainer" containerID="01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186" Nov 03 20:31:48.293400 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.293380 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186\": container with ID starting with 01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186 not found: ID does not exist" containerID="01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186" Nov 03 20:31:48.293481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.293413 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186"} err="failed to get container status \"01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186\": rpc error: code = NotFound desc = could not find container \"01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186\": container with ID starting with 01c45295d599d2d82d839cbd9e0cf2f43c8cc6a2279810083e6ae27627011186 not found: ID does not exist" Nov 03 20:31:48.305415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.305389 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v"] Nov 03 20:31:48.305965 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.305938 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-b0ef96\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:48.308353 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.308325 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-e20206\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:48.310563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.310498 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-7a92d4/auth-test-model-7a92d4-kserve-65f7db5545-4955v"] Nov 03 20:31:48.560757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.560734 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:48.563544 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.563515 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-b0ef96\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" pod="opendatahub/test-curl-7567-b0ef96" Nov 03 20:31:48.565802 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:48.565773 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-e20206\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:48.589724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.589696 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" path="/var/lib/kubelet/pods/2592b7c8-8386-4df0-8509-caf805154d4c/volumes" Nov 03 20:31:48.590280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.590259 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c9dea9-da8b-4ad8-a06f-0df2b14f2e88" path="/var/lib/kubelet/pods/25c9dea9-da8b-4ad8-a06f-0df2b14f2e88/volumes" Nov 03 20:31:48.600925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.600898 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca\") pod \"b008bd53-0d87-4f40-890f-1ab3384913d8\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " Nov 03 20:31:48.601106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.601051 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l\" (UniqueName: \"kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l\") pod \"b008bd53-0d87-4f40-890f-1ab3384913d8\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " Nov 03 20:31:48.601736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.601659 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca\") pod \"b008bd53-0d87-4f40-890f-1ab3384913d8\" (UID: \"b008bd53-0d87-4f40-890f-1ab3384913d8\") " Nov 03 20:31:48.602353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.602327 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca" pod "b008bd53-0d87-4f40-890f-1ab3384913d8" (UID: "b008bd53-0d87-4f40-890f-1ab3384913d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:48.602671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.602626 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:48.604897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.604863 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l" pod "b008bd53-0d87-4f40-890f-1ab3384913d8" (UID: "b008bd53-0d87-4f40-890f-1ab3384913d8"). InnerVolumeSpecName "kube-api-access-p292l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:48.605015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.604992 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca" pod "b008bd53-0d87-4f40-890f-1ab3384913d8" (UID: "b008bd53-0d87-4f40-890f-1ab3384913d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:48.704135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.704104 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b008bd53-0d87-4f40-890f-1ab3384913d8-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:48.704286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.704138 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p292l\" (UniqueName: \"kubernetes.io/projected/b008bd53-0d87-4f40-890f-1ab3384913d8-kube-api-access-p292l\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:48.993978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.993943 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-3041f7"] Nov 03 20:31:48.999149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:48.999118 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-3041f7"] Nov 03 20:31:49.279163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:49.279078 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" event={"ID":"b008bd53-0d87-4f40-890f-1ab3384913d8","Type":"ContainerDied","Data":"6e32c6a5474ffff4bd54464ae0e76375d35fead9b584a392c0246853c820fed6"} Nov 03 20:31:49.279163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:49.279153 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e32c6a5474ffff4bd54464ae0e76375d35fead9b584a392c0246853c820fed6" Nov 03 20:31:49.279163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:49.279083 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9" Nov 03 20:31:50.037999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.037971 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:31:50.088705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088257 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.088705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088320 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.088705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088415 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.088705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088468 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.088705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088494 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.088705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088565 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.090352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.088658 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.090352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.089196 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.090352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.089225 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data\") pod \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\" (UID: \"b19cd843-43af-48c3-b4a8-8d268bfd5f35\") " Nov 03 20:31:50.090687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.090663 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.093732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.093698 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.093908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.093890 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.100566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.095999 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.100566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.098708 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.100566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.099850 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "kube-api-access-h4sh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.100885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.100862 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.101288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.101015 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:31:50.101288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.101101 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.101640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.101615 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs" pod "b19cd843-43af-48c3-b4a8-8d268bfd5f35" (UID: "b19cd843-43af-48c3-b4a8-8d268bfd5f35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.113685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.113659 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:31:50.191298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.190976 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.191298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191055 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.191298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191086 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.191298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191209 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.191298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191244 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.191712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191310 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.191712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191459 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.191712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191546 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.191712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191581 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.191712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191659 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.191712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191698 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191736 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191787 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191827 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191878 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191922 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data\") pod \"76d4b78b-0901-41a6-b416-80856daade9c\" (UID: \"76d4b78b-0901-41a6-b416-80856daade9c\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.191957 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.192140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.192002 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert\") pod \"406afe59-8851-4178-8e7e-fad68db839ad\" (UID: \"406afe59-8851-4178-8e7e-fad68db839ad\") " Nov 03 20:31:50.194610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.193772 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.194610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.193901 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.194610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.194105 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.194610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.194324 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.195165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.195143 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.195565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.195544 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.195819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.195791 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.196467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.196363 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.196604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.196582 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.197298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.196883 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.197469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197242 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197698 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197741 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197754 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197767 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197781 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197795 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/406afe59-8851-4178-8e7e-fad68db839ad-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197808 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4sh5\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-kube-api-access-h4sh5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197823 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197835 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197849 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d4b78b-0901-41a6-b416-80856daade9c-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197862 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197875 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197889 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197902 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197916 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19cd843-43af-48c3-b4a8-8d268bfd5f35-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197930 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197945 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.197968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.197961 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19cd843-43af-48c3-b4a8-8d268bfd5f35-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.199540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.199255 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.199540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.199347 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.199709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.199688 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.199784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.199715 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "kube-api-access-2zxwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.199936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.199797 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.200821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.200678 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.200821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.200796 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token" pod "76d4b78b-0901-41a6-b416-80856daade9c" (UID: "76d4b78b-0901-41a6-b416-80856daade9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.201691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.201669 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v" pod "406afe59-8851-4178-8e7e-fad68db839ad" (UID: "406afe59-8851-4178-8e7e-fad68db839ad"). InnerVolumeSpecName "kube-api-access-xsj5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.290593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.290566 5815 generic.go:356] "Generic (PLEG): container finished" podID="76d4b78b-0901-41a6-b416-80856daade9c" containerID="42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060" exitCode=0 Nov 03 20:31:50.291024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.290622 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" Nov 03 20:31:50.291024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.290647 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" event={"ID":"76d4b78b-0901-41a6-b416-80856daade9c","Type":"ContainerDied","Data":"42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060"} Nov 03 20:31:50.291024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.290705 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5" event={"ID":"76d4b78b-0901-41a6-b416-80856daade9c","Type":"ContainerDied","Data":"778b597b2a3c33ee5a3bd4fccd63e9cb47f6e4c7f2bc137a399d3e41cfeea4d1"} Nov 03 20:31:50.291024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.290729 5815 scope.go:122] "RemoveContainer" containerID="42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060" Nov 03 20:31:50.293118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.293093 5815 generic.go:356] "Generic (PLEG): container finished" podID="406afe59-8851-4178-8e7e-fad68db839ad" containerID="2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a" exitCode=0 Nov 03 20:31:50.293217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.293134 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" Nov 03 20:31:50.293217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.293171 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" event={"ID":"406afe59-8851-4178-8e7e-fad68db839ad","Type":"ContainerDied","Data":"2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a"} Nov 03 20:31:50.293217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.293212 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8" event={"ID":"406afe59-8851-4178-8e7e-fad68db839ad","Type":"ContainerDied","Data":"f94443d65573b9e88dce4d61dd08e96d3c808eeae3d068106f25e6ef4dc47c87"} Nov 03 20:31:50.297624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.295058 5815 generic.go:356] "Generic (PLEG): container finished" podID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerID="e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8" exitCode=0 Nov 03 20:31:50.297624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.295117 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" Nov 03 20:31:50.297624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.295155 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" event={"ID":"b19cd843-43af-48c3-b4a8-8d268bfd5f35","Type":"ContainerDied","Data":"e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8"} Nov 03 20:31:50.297624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.295195 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft" event={"ID":"b19cd843-43af-48c3-b4a8-8d268bfd5f35","Type":"ContainerDied","Data":"b6cf222b8eb9691a6930b5870af92b1fe9a1862ab11775c504eaea9c3a4cf1da"} Nov 03 20:31:50.297624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.295407 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7567-3041f7" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" containerName="test-curl-7567-3041f7" containerID="cri-o://c691e3f0c014f6f8235f4c5d15e78c188900479a4dbc50f03253e4296c5f26db" gracePeriod=2 Nov 03 20:31:50.299362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299337 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d4b78b-0901-41a6-b416-80856daade9c-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299367 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299382 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299395 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsj5v\" (UniqueName: \"kubernetes.io/projected/406afe59-8851-4178-8e7e-fad68db839ad-kube-api-access-xsj5v\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299410 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/406afe59-8851-4178-8e7e-fad68db839ad-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299440 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/406afe59-8851-4178-8e7e-fad68db839ad-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299454 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxwr\" (UniqueName: \"kubernetes.io/projected/76d4b78b-0901-41a6-b416-80856daade9c-kube-api-access-2zxwr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.299464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.299467 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d4b78b-0901-41a6-b416-80856daade9c-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.304498 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.304475 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:50.326954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.326937 5815 scope.go:122] "RemoveContainer" containerID="42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060" Nov 03 20:31:50.327274 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.327253 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060\": container with ID starting with 42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060 not found: ID does not exist" containerID="42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060" Nov 03 20:31:50.327379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.327282 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060"} err="failed to get container status \"42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060\": rpc error: code = NotFound desc = could not find container \"42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060\": container with ID starting with 42480212cced8c436b69d0cb5272c7123d2827bccc4b9b25bd30c79d9311c060 not found: ID does not exist" Nov 03 20:31:50.327379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.327303 5815 scope.go:122] "RemoveContainer" containerID="2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a" Nov 03 20:31:50.340895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.340868 5815 scope.go:122] "RemoveContainer" containerID="441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183" Nov 03 20:31:50.346129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.346105 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8"] Nov 03 20:31:50.349756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.349736 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-hxbd8"] Nov 03 20:31:50.361690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.361128 5815 scope.go:122] "RemoveContainer" containerID="2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a" Nov 03 20:31:50.366037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.365519 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5"] Nov 03 20:31:50.366037 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.365699 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:50.369848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.369823 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fn4t5"] Nov 03 20:31:50.371026 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.370998 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a\": container with ID starting with 2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a not found: ID does not exist" containerID="2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a" Nov 03 20:31:50.371098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.371038 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a"} err="failed to get container status \"2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a\": rpc error: code = NotFound desc = could not find container \"2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a\": container with ID starting with 2a8b122d0f6b11c6debd31fdec0c840d0a1629f596b7fd700094bec6e1d6ec3a not found: ID does not exist" Nov 03 20:31:50.371098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.371062 5815 scope.go:122] "RemoveContainer" containerID="441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183" Nov 03 20:31:50.371443 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.371415 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183\": container with ID starting with 441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183 not found: ID does not exist" containerID="441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183" Nov 03 20:31:50.371486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.371452 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183"} err="failed to get container status \"441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183\": rpc error: code = NotFound desc = could not find container \"441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183\": container with ID starting with 441f15ffd4de063a6586e24ef7faaacf9e033703152cb5901672a80c9dd38183 not found: ID does not exist" Nov 03 20:31:50.371486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.371473 5815 scope.go:122] "RemoveContainer" containerID="e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8" Nov 03 20:31:50.384424 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.384395 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:50.385616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.385599 5815 scope.go:122] "RemoveContainer" containerID="e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8" Nov 03 20:31:50.385952 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.385892 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8\": container with ID starting with e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8 not found: ID does not exist" containerID="e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8" Nov 03 20:31:50.385952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.385925 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8"} err="failed to get container status \"e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8\": rpc error: code = NotFound desc = could not find container \"e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8\": container with ID starting with e3a7cc9be9f3db11df07717a89dadb95feffc264eee6c4c9ac6f252c304127b8 not found: ID does not exist" Nov 03 20:31:50.386268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.386248 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft"] Nov 03 20:31:50.392336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.392312 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-5hbft"] Nov 03 20:31:50.458627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.458603 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:50.461260 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.461226 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:50.639522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.639382 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz\" (UniqueName: \"kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz\") pod \"3295711e-1416-4bcd-9e12-2b44f1931dcf\" (UID: \"3295711e-1416-4bcd-9e12-2b44f1931dcf\") " Nov 03 20:31:50.672046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.672014 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz" pod "3295711e-1416-4bcd-9e12-2b44f1931dcf" (UID: "3295711e-1416-4bcd-9e12-2b44f1931dcf"). InnerVolumeSpecName "kube-api-access-dsvcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.675140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.675104 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3295711e-1416-4bcd-9e12-2b44f1931dcf" path="/var/lib/kubelet/pods/3295711e-1416-4bcd-9e12-2b44f1931dcf/volumes" Nov 03 20:31:50.675737 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:50.675707 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:50.675926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.675821 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406afe59-8851-4178-8e7e-fad68db839ad" path="/var/lib/kubelet/pods/406afe59-8851-4178-8e7e-fad68db839ad/volumes" Nov 03 20:31:50.676919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.676899 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d4b78b-0901-41a6-b416-80856daade9c" path="/var/lib/kubelet/pods/76d4b78b-0901-41a6-b416-80856daade9c/volumes" Nov 03 20:31:50.677806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.677788 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" path="/var/lib/kubelet/pods/b19cd843-43af-48c3-b4a8-8d268bfd5f35/volumes" Nov 03 20:31:50.758432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:50.758400 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsvcz\" (UniqueName: \"kubernetes.io/projected/3295711e-1416-4bcd-9e12-2b44f1931dcf-kube-api-access-dsvcz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:51.468544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:51.465861 5815 generic.go:356] "Generic (PLEG): container finished" podID="3295711e-1416-4bcd-9e12-2b44f1931dcf" containerID="b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d" exitCode=137 Nov 03 20:31:51.468544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:51.466075 5815 scope.go:122] "RemoveContainer" containerID="b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d" Nov 03 20:31:51.468544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:51.466292 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-e20206" Nov 03 20:31:51.651535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:51.649645 5815 scope.go:122] "RemoveContainer" containerID="b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d" Nov 03 20:31:51.670577 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:51.667337 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d\": container with ID starting with b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d not found: ID does not exist" containerID="b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d" Nov 03 20:31:51.670577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:51.667376 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d"} err="failed to get container status \"b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d\": rpc error: code = NotFound desc = could not find container \"b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d\": container with ID starting with b750e6d0ad953b2ed645e6c357ab3343d97f268404c1dbdf5e90c046d23e472d not found: ID does not exist" Nov 03 20:31:51.881060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:51.880952 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" containerID="cri-o://11170986b9a4ce745ae075f55cff68f3e9b9199eb158a4c51c1619d19aa9665d" gracePeriod=45 Nov 03 20:31:52.118767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.116128 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.209:8000/ready\": dial tcp 10.132.0.209:8000: connect: connection refused" Nov 03 20:31:52.491914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.491770 5815 generic.go:356] "Generic (PLEG): container finished" podID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" containerID="c691e3f0c014f6f8235f4c5d15e78c188900479a4dbc50f03253e4296c5f26db" exitCode=137 Nov 03 20:31:52.494465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.494296 5815 generic.go:356] "Generic (PLEG): container finished" podID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerID="11170986b9a4ce745ae075f55cff68f3e9b9199eb158a4c51c1619d19aa9665d" exitCode=0 Nov 03 20:31:52.494465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.494424 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" event={"ID":"f8df9da6-4c0d-43c4-9656-2b64fc448deb","Type":"ContainerDied","Data":"11170986b9a4ce745ae075f55cff68f3e9b9199eb158a4c51c1619d19aa9665d"} Nov 03 20:31:52.660395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.660360 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:52.727234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.727180 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq\" (UniqueName: \"kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq\") pod \"4c2fd8be-ad57-4cd5-b079-adfd0141054b\" (UID: \"4c2fd8be-ad57-4cd5-b079-adfd0141054b\") " Nov 03 20:31:52.729670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.729629 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq" pod "4c2fd8be-ad57-4cd5-b079-adfd0141054b" (UID: "4c2fd8be-ad57-4cd5-b079-adfd0141054b"). InnerVolumeSpecName "kube-api-access-slpbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:52.967727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:52.966150 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slpbq\" (UniqueName: \"kubernetes.io/projected/4c2fd8be-ad57-4cd5-b079-adfd0141054b-kube-api-access-slpbq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.034169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.034143 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:31:53.067161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067129 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm\") pod \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " Nov 03 20:31:53.067379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067282 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir\") pod \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " Nov 03 20:31:53.067458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067415 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx\" (UniqueName: \"kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx\") pod \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " Nov 03 20:31:53.067534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067473 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home\") pod \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " Nov 03 20:31:53.067610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067560 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs\") pod \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " Nov 03 20:31:53.067610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067594 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir" pod "f8df9da6-4c0d-43c4-9656-2b64fc448deb" (UID: "f8df9da6-4c0d-43c4-9656-2b64fc448deb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:53.067741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067644 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache\") pod \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\" (UID: \"f8df9da6-4c0d-43c4-9656-2b64fc448deb\") " Nov 03 20:31:53.067828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067809 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home" pod "f8df9da6-4c0d-43c4-9656-2b64fc448deb" (UID: "f8df9da6-4c0d-43c4-9656-2b64fc448deb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:53.068000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.067982 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache" pod "f8df9da6-4c0d-43c4-9656-2b64fc448deb" (UID: "f8df9da6-4c0d-43c4-9656-2b64fc448deb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:53.068274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.068258 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.068347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.068279 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.068347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.068291 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.071848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.071818 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs" pod "f8df9da6-4c0d-43c4-9656-2b64fc448deb" (UID: "f8df9da6-4c0d-43c4-9656-2b64fc448deb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:53.071974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.071956 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx" pod "f8df9da6-4c0d-43c4-9656-2b64fc448deb" (UID: "f8df9da6-4c0d-43c4-9656-2b64fc448deb"). InnerVolumeSpecName "kube-api-access-nqmwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:53.073476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.073417 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm" pod "f8df9da6-4c0d-43c4-9656-2b64fc448deb" (UID: "f8df9da6-4c0d-43c4-9656-2b64fc448deb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:53.170463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.170423 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8df9da6-4c0d-43c4-9656-2b64fc448deb-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.170697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.170465 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmwx\" (UniqueName: \"kubernetes.io/projected/f8df9da6-4c0d-43c4-9656-2b64fc448deb-kube-api-access-nqmwx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.170697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.170483 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8df9da6-4c0d-43c4-9656-2b64fc448deb-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.615103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.613551 5815 scope.go:122] "RemoveContainer" containerID="c691e3f0c014f6f8235f4c5d15e78c188900479a4dbc50f03253e4296c5f26db" Nov 03 20:31:53.615103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.613723 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:53.664649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.660362 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" event={"ID":"f8df9da6-4c0d-43c4-9656-2b64fc448deb","Type":"ContainerDied","Data":"77a83550c73d09c07ee2a77b1cb5ddd44d5161574ff13f6f4555cdf39ea7d3cd"} Nov 03 20:31:53.664649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.660472 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787" Nov 03 20:31:53.787312 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:53.780239 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:53.807752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:53.806230 5815 scope.go:122] "RemoveContainer" containerID="11170986b9a4ce745ae075f55cff68f3e9b9199eb158a4c51c1619d19aa9665d" Nov 03 20:31:53.954570 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:53.954498 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:54.004893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:54.004828 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787"] Nov 03 20:31:54.052103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:54.052036 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-08a7bc/auth-test-model-08a7bc-kserve-6fdf6f87fc-vm787"] Nov 03 20:31:54.229788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:54.229757 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw"] Nov 03 20:31:54.232741 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:31:54.232709 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7567-3041f7\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" pod="opendatahub/test-curl-7567-3041f7" Nov 03 20:31:54.632824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:54.630773 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" path="/var/lib/kubelet/pods/4c2fd8be-ad57-4cd5-b079-adfd0141054b/volumes" Nov 03 20:31:54.634177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:54.634142 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" path="/var/lib/kubelet/pods/f8df9da6-4c0d-43c4-9656-2b64fc448deb/volumes" Nov 03 20:31:54.661312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:54.661269 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl"] Nov 03 20:31:56.201699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.201658 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn"] Nov 03 20:31:56.203048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203016 5815 cpu_manager.go:414] "containerMap: removing container" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" Nov 03 20:31:56.203048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203058 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203078 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b008bd53-0d87-4f40-890f-1ab3384913d8" containerName="revoke-keys" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203085 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b008bd53-0d87-4f40-890f-1ab3384913d8" containerName="revoke-keys" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203095 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" containerName="test-curl-7567-3041f7" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203104 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" containerName="test-curl-7567-3041f7" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203116 5815 cpu_manager.go:414] "containerMap: removing container" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203123 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203143 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203151 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203197 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203205 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203248 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" Nov 03 20:31:56.203271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203257 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203450 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8df9da6-4c0d-43c4-9656-2b64fc448deb" containerName="main" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203470 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203484 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d4b78b-0901-41a6-b416-80856daade9c" containerName="istio-proxy" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203493 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b008bd53-0d87-4f40-890f-1ab3384913d8" containerName="revoke-keys" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203526 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2fd8be-ad57-4cd5-b079-adfd0141054b" containerName="test-curl-7567-3041f7" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203536 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2592b7c8-8386-4df0-8509-caf805154d4c" containerName="main" Nov 03 20:31:56.204067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.203556 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19cd843-43af-48c3-b4a8-8d268bfd5f35" containerName="istio-proxy" Nov 03 20:31:56.222741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.210421 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.222741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.217887 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn"] Nov 03 20:31:56.299317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.299269 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhj6p\" (UniqueName: \"kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p\") pod \"payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.299317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.299323 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.304958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.304907 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-dce009-dockercfg-kqglx\"" Nov 03 20:31:56.305851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.305823 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-dce009\"" Nov 03 20:31:56.416529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.410085 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhj6p\" (UniqueName: \"kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p\") pod \"payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.416529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.410142 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.416529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.411538 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.467590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.461575 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhj6p\" (UniqueName: \"kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p\") pod \"payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:56.551538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:56.549792 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:57.052430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:57.035012 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn"] Nov 03 20:31:57.052430 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:31:57.040577 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca54681f_3a06_45ef_ab20_5d0607c178dc.slice/crio-c224669e213cafc4f40f82c2ccab797a5a522bc9cdcb7bf4212f3bd1212d91e6 WatchSource:0}: Error finding container c224669e213cafc4f40f82c2ccab797a5a522bc9cdcb7bf4212f3bd1212d91e6: Status 404 returned error can't find the container with id c224669e213cafc4f40f82c2ccab797a5a522bc9cdcb7bf4212f3bd1212d91e6 Nov 03 20:31:57.730915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:57.730882 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" event={"ID":"ca54681f-3a06-45ef-ab20-5d0607c178dc","Type":"ContainerStarted","Data":"14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d"} Nov 03 20:31:57.731341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:57.730933 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" event={"ID":"ca54681f-3a06-45ef-ab20-5d0607c178dc","Type":"ContainerStarted","Data":"c224669e213cafc4f40f82c2ccab797a5a522bc9cdcb7bf4212f3bd1212d91e6"} Nov 03 20:31:57.731341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:57.730991 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:31:57.754964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:57.754901 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" podStartSLOduration=1.754881849 podStartE2EDuration="1.754881849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:57.75177902 +0000 UTC m=+1827.282124693" watchObservedRunningTime="2026-11-03 20:31:57.754881849 +0000 UTC m=+1827.285227547" Nov 03 20:31:59.082531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:31:59.068346 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5"] Nov 03 20:32:01.113775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.113736 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp"] Nov 03 20:32:01.114909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.114879 5815 cpu_manager.go:414] "containerMap: removing container" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" Nov 03 20:32:01.114909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.114910 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" Nov 03 20:32:01.115143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.115118 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="406afe59-8851-4178-8e7e-fad68db839ad" containerName="istio-proxy" Nov 03 20:32:01.119964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.119942 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.145157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.145129 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp"] Nov 03 20:32:01.156816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.156790 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0f6e09-openshift-default-dockercfg-ksg7s\"" Nov 03 20:32:01.225248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225213 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225342 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225381 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225539 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225602 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225718 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225763 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225881 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.225968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.225930 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgq9d\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.336942 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgq9d\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.337489 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.338041 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.338320 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.338395 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.339064 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.339144 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.339435 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.339844 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.340141 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.340242 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.340342 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.343534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.342150 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.354606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.345150 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.375585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.366036 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.375585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.374814 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.375585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.375541 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgq9d\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.382532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.375996 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token\") pod \"e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.459536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.458809 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:01.802532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:01.801721 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc"] Nov 03 20:32:02.451529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.451293 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j"] Nov 03 20:32:02.603643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.476486 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp"] Nov 03 20:32:02.603643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.520127 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:02.603643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.600837 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:02.603643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.600900 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:02.889533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.880970 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv"] Nov 03 20:32:02.898924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.898899 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:02.914528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.903772 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ed5ad5-openshift-default-dockercfg-7jph6\"" Nov 03 20:32:02.932321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.932267 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv"] Nov 03 20:32:02.937828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.937801 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" event={"ID":"332403f8-ebc6-4a19-be4f-ea8a58402fd0","Type":"ContainerStarted","Data":"f22ca6498220400941f74c0a4842985e1112446625e50d50bcbceab2dcaff954"} Nov 03 20:32:02.939186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:02.939161 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" event={"ID":"332403f8-ebc6-4a19-be4f-ea8a58402fd0","Type":"ContainerStarted","Data":"2b3d9ec893b4690a77ee693edfecbf0f117174ddc90218130fae730a98d6e0c5"} Nov 03 20:32:03.148536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.142218 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podStartSLOduration=2.142198012 podStartE2EDuration="2.142198012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:03.100263268 +0000 UTC m=+1832.630608941" watchObservedRunningTime="2026-11-03 20:32:03.142198012 +0000 UTC m=+1832.672543686" Nov 03 20:32:03.208951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.208856 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.209225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.209203 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.209362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.209347 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.209600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.209581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.209814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.209795 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw449\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.209929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.209915 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.210048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.210033 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.210212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.210197 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.210395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.210379 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.313113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.313068 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.314136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.314110 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw449\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.314680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.314660 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.319004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.318983 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.319545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.319441 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.320300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.320277 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.330092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.330066 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.330287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.319810 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.330530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.330488 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.330894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.330874 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.331089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.330953 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.331558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.331259 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.331725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.331708 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.345281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.345249 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.347285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.347254 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.347436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.332666 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.362571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.349628 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw449\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.402066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.393278 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token\") pod \"e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:03.500535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.499614 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:03.501077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.500950 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:32:03.501077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.500998 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:32:03.560546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:03.557857 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:04.077540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.077302 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:04.107500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.107462 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:04.124544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.123718 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:32:04.142071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.142029 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/health\": context deadline exceeded" Nov 03 20:32:04.245916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.245858 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:04.262332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.261950 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:04.263355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.262988 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:04.460670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.460644 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:32:04.460919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.460823 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:32:04.660285 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:04.660241 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e264e86_9138_4e30_8dc1_90aee8ba7746.slice/crio-4921ecb76b724aad704c8c0f550b03449f7d47e59607d28d9a00467fc536fffe WatchSource:0}: Error finding container 4921ecb76b724aad704c8c0f550b03449f7d47e59607d28d9a00467fc536fffe: Status 404 returned error can't find the container with id 4921ecb76b724aad704c8c0f550b03449f7d47e59607d28d9a00467fc536fffe Nov 03 20:32:04.660781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.660443 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv"] Nov 03 20:32:04.663911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.663876 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:04.664024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.663953 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:04.664096 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:04.664020 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:05.203004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:05.202964 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" event={"ID":"4e264e86-9138-4e30-8dc1-90aee8ba7746","Type":"ContainerStarted","Data":"dadd6f4c9c1b14e716ae008803c10576d51ec1d030de756b544ec2ea49d384b4"} Nov 03 20:32:05.203220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:05.203027 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" event={"ID":"4e264e86-9138-4e30-8dc1-90aee8ba7746","Type":"ContainerStarted","Data":"4921ecb76b724aad704c8c0f550b03449f7d47e59607d28d9a00467fc536fffe"} Nov 03 20:32:05.237560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:05.235420 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" podStartSLOduration=3.235403149 podStartE2EDuration="3.235403149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:05.229408418 +0000 UTC m=+1834.759754092" watchObservedRunningTime="2026-11-03 20:32:05.235403149 +0000 UTC m=+1834.765748822" Nov 03 20:32:05.464933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:05.464903 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:05.567754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:05.567722 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:05.867941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:05.867862 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:06.213669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:06.213638 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:06.213886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:06.213699 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:06.214679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:06.214648 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:06.214827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:06.214726 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:32:08.746727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:08.746686 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 20:32:09.375792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:09.375744 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" containerID="cri-o://9bfcfef7f975db1f7b6ffbf555054fdeab4f883ab25e938202c53c075f9b615e" gracePeriod=45 Nov 03 20:32:09.705004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:09.704970 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": dial tcp 10.132.0.215:8000: connect: connection refused" Nov 03 20:32:09.741396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:09.741347 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" containerID="cri-o://676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115" gracePeriod=45 Nov 03 20:32:09.868731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:09.868699 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.213:8000/ready\": dial tcp 10.132.0.213:8000: connect: connection refused" Nov 03 20:32:10.071258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.071230 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.150610 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm\") pod \"636da0b1-49bc-4139-9f5f-111384502bd1\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.150862 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache\") pod \"636da0b1-49bc-4139-9f5f-111384502bd1\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.150900 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth\" (UniqueName: \"kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth\") pod \"636da0b1-49bc-4139-9f5f-111384502bd1\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.151169 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache" pod "636da0b1-49bc-4139-9f5f-111384502bd1" (UID: "636da0b1-49bc-4139-9f5f-111384502bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.151253 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir\") pod \"636da0b1-49bc-4139-9f5f-111384502bd1\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.151290 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs\" (UniqueName: \"kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs\") pod \"636da0b1-49bc-4139-9f5f-111384502bd1\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.151615 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir" pod "636da0b1-49bc-4139-9f5f-111384502bd1" (UID: "636da0b1-49bc-4139-9f5f-111384502bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.151860 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home" pod "636da0b1-49bc-4139-9f5f-111384502bd1" (UID: "636da0b1-49bc-4139-9f5f-111384502bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:10.152568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.151350 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home\") pod \"636da0b1-49bc-4139-9f5f-111384502bd1\" (UID: \"636da0b1-49bc-4139-9f5f-111384502bd1\") " Nov 03 20:32:10.153274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.152665 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.153274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.152687 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.153274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.152701 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.159839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.159188 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm" pod "636da0b1-49bc-4139-9f5f-111384502bd1" (UID: "636da0b1-49bc-4139-9f5f-111384502bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:10.162830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.162799 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs" pod "636da0b1-49bc-4139-9f5f-111384502bd1" (UID: "636da0b1-49bc-4139-9f5f-111384502bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:10.162978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.162945 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth" pod "636da0b1-49bc-4139-9f5f-111384502bd1" (UID: "636da0b1-49bc-4139-9f5f-111384502bd1"). InnerVolumeSpecName "kube-api-access-6zfth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:10.258536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.255429 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8"] Nov 03 20:32:10.258536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.255446 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zfth\" (UniqueName: \"kubernetes.io/projected/636da0b1-49bc-4139-9f5f-111384502bd1-kube-api-access-6zfth\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.258536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.255610 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/636da0b1-49bc-4139-9f5f-111384502bd1-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.258536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.255624 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/636da0b1-49bc-4139-9f5f-111384502bd1-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.433534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.422225 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7564-e4c772"] Nov 03 20:32:10.433534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.423459 5815 cpu_manager.go:414] "containerMap: removing container" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" Nov 03 20:32:10.433534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.423482 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" Nov 03 20:32:10.433534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.423692 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" containerName="main" Nov 03 20:32:10.507188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.507137 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-e4c772"] Nov 03 20:32:10.507425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.507373 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:10.672999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.656475 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6p5pq\"" Nov 03 20:32:10.672999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.664684 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qskk\" (UniqueName: \"kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk\") pod \"test-curl-7564-e4c772\" (UID: \"ba71efa8-290a-4a65-a976-bb9fae54ca16\") " pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:10.705629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.705594 5815 generic.go:356] "Generic (PLEG): container finished" podID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerID="9bfcfef7f975db1f7b6ffbf555054fdeab4f883ab25e938202c53c075f9b615e" exitCode=0 Nov 03 20:32:10.707565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.707539 5815 generic.go:356] "Generic (PLEG): container finished" podID="636da0b1-49bc-4139-9f5f-111384502bd1" containerID="676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115" exitCode=0 Nov 03 20:32:10.708044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.707630 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" Nov 03 20:32:10.764041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.763935 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" event={"ID":"744520ef-cd92-40a9-a060-c97a0af7c33e","Type":"ContainerDied","Data":"9bfcfef7f975db1f7b6ffbf555054fdeab4f883ab25e938202c53c075f9b615e"} Nov 03 20:32:10.764741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.764712 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" event={"ID":"636da0b1-49bc-4139-9f5f-111384502bd1","Type":"ContainerDied","Data":"676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115"} Nov 03 20:32:10.764920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.764906 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl" event={"ID":"636da0b1-49bc-4139-9f5f-111384502bd1","Type":"ContainerDied","Data":"1d0618feff07f9afadd44a3a4505b4bfec221740f3a11e3ac5e413d6ea405051"} Nov 03 20:32:10.765046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.765032 5815 scope.go:122] "RemoveContainer" containerID="676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115" Nov 03 20:32:10.787922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.787851 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl"] Nov 03 20:32:10.788761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.788733 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3c25bf/auth-test-model-3c25bf-kserve-dd8c5c5b9-zg8nl"] Nov 03 20:32:10.884751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.884724 5815 scope.go:122] "RemoveContainer" containerID="676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115" Nov 03 20:32:10.886080 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:10.886056 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115\": container with ID starting with 676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115 not found: ID does not exist" containerID="676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115" Nov 03 20:32:10.936782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.927482 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115"} err="failed to get container status \"676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115\": rpc error: code = NotFound desc = could not find container \"676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115\": container with ID starting with 676ffcea8a040d86e2abb82bf518a01b2dde73be307edf6aa0b1bf1dfce4e115 not found: ID does not exist" Nov 03 20:32:10.936782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:10.888998 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qskk\" (UniqueName: \"kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk\") pod \"test-curl-7564-e4c772\" (UID: \"ba71efa8-290a-4a65-a976-bb9fae54ca16\") " pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:11.040674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.040639 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qskk\" (UniqueName: \"kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk\") pod \"test-curl-7564-e4c772\" (UID: \"ba71efa8-290a-4a65-a976-bb9fae54ca16\") " pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:11.213518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.213478 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:32:11.370056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.368806 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:11.396970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.396928 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache\") pod \"744520ef-cd92-40a9-a060-c97a0af7c33e\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " Nov 03 20:32:11.397199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.397036 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home\") pod \"744520ef-cd92-40a9-a060-c97a0af7c33e\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " Nov 03 20:32:11.397307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.397277 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw\" (UniqueName: \"kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw\") pod \"744520ef-cd92-40a9-a060-c97a0af7c33e\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " Nov 03 20:32:11.397377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.397313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs\" (UniqueName: \"kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs\") pod \"744520ef-cd92-40a9-a060-c97a0af7c33e\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " Nov 03 20:32:11.397377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.397367 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm\") pod \"744520ef-cd92-40a9-a060-c97a0af7c33e\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " Nov 03 20:32:11.397552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.397531 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir\") pod \"744520ef-cd92-40a9-a060-c97a0af7c33e\" (UID: \"744520ef-cd92-40a9-a060-c97a0af7c33e\") " Nov 03 20:32:11.426817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.415746 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache" pod "744520ef-cd92-40a9-a060-c97a0af7c33e" (UID: "744520ef-cd92-40a9-a060-c97a0af7c33e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:11.426817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.419606 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir" pod "744520ef-cd92-40a9-a060-c97a0af7c33e" (UID: "744520ef-cd92-40a9-a060-c97a0af7c33e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:11.507577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.506262 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm" pod "744520ef-cd92-40a9-a060-c97a0af7c33e" (UID: "744520ef-cd92-40a9-a060-c97a0af7c33e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:11.515891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.508156 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.515891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.508187 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.515891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.508202 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.515891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.514717 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw" pod "744520ef-cd92-40a9-a060-c97a0af7c33e" (UID: "744520ef-cd92-40a9-a060-c97a0af7c33e"). InnerVolumeSpecName "kube-api-access-wdpgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:11.546301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.546246 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs" pod "744520ef-cd92-40a9-a060-c97a0af7c33e" (UID: "744520ef-cd92-40a9-a060-c97a0af7c33e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:11.578544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.576873 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home" pod "744520ef-cd92-40a9-a060-c97a0af7c33e" (UID: "744520ef-cd92-40a9-a060-c97a0af7c33e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:11.680528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.645160 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744520ef-cd92-40a9-a060-c97a0af7c33e-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.680528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.645198 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdpgw\" (UniqueName: \"kubernetes.io/projected/744520ef-cd92-40a9-a060-c97a0af7c33e-kube-api-access-wdpgw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.680528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.645214 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744520ef-cd92-40a9-a060-c97a0af7c33e-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.705561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.702831 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:11.705561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.702944 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:32:11.730224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.728167 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" event={"ID":"744520ef-cd92-40a9-a060-c97a0af7c33e","Type":"ContainerDied","Data":"7763715c7f1d7adee965c88761cd7d6dae77a979c274041717e750f9f86e5af6"} Nov 03 20:32:11.730224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.728233 5815 scope.go:122] "RemoveContainer" containerID="9bfcfef7f975db1f7b6ffbf555054fdeab4f883ab25e938202c53c075f9b615e" Nov 03 20:32:11.730224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.728346 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw" Nov 03 20:32:11.781864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.766746 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw"] Nov 03 20:32:11.781864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:11.775954 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/auth-test-model-bab2c3-kserve-5fbdcd5985-tjwgw"] Nov 03 20:32:11.987972 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:11.987156 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba71efa8_290a_4a65_a976_bb9fae54ca16.slice/crio-5edd30fc5d23ede443b21a059945d1828cd33380b2f18d909c8fc55b4c3565f1 WatchSource:0}: Error finding container 5edd30fc5d23ede443b21a059945d1828cd33380b2f18d909c8fc55b4c3565f1: Status 404 returned error can't find the container with id 5edd30fc5d23ede443b21a059945d1828cd33380b2f18d909c8fc55b4c3565f1 Nov 03 20:32:12.009181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.009121 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7564-e4c772"] Nov 03 20:32:12.591349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.591318 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636da0b1-49bc-4139-9f5f-111384502bd1" path="/var/lib/kubelet/pods/636da0b1-49bc-4139-9f5f-111384502bd1/volumes" Nov 03 20:32:12.592045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.592016 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" path="/var/lib/kubelet/pods/744520ef-cd92-40a9-a060-c97a0af7c33e/volumes" Nov 03 20:32:12.655479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.655442 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:32:12.744973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.744935 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-e4c772" event={"ID":"ba71efa8-290a-4a65-a976-bb9fae54ca16","Type":"ContainerStarted","Data":"aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a"} Nov 03 20:32:12.745153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.744987 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7564-e4c772" event={"ID":"ba71efa8-290a-4a65-a976-bb9fae54ca16","Type":"ContainerStarted","Data":"5edd30fc5d23ede443b21a059945d1828cd33380b2f18d909c8fc55b4c3565f1"} Nov 03 20:32:12.763260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:12.763215 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7564-e4c772" podStartSLOduration=2.763201509 podStartE2EDuration="2.763201509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:12.760556815 +0000 UTC m=+1842.290902487" watchObservedRunningTime="2026-11-03 20:32:12.763201509 +0000 UTC m=+1842.293547181" Nov 03 20:32:14.152542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.151871 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" containerID="cri-o://58f9322fc4903e3e0844a1a7796430f1607e64b7128dd2663aefde73fea22663" gracePeriod=45 Nov 03 20:32:14.224674 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:14.213774 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:32:14.263536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.262651 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": read tcp 10.132.0.2:37768->10.132.0.214:8000: read: connection reset by peer" Nov 03 20:32:14.484531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.482612 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:14.681784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.661888 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:14.797883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.797791 5815 generic.go:356] "Generic (PLEG): container finished" podID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerID="58f9322fc4903e3e0844a1a7796430f1607e64b7128dd2663aefde73fea22663" exitCode=0 Nov 03 20:32:14.798070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.797957 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" event={"ID":"279ab75f-c9e2-4d5a-9907-a608f7633b7a","Type":"ContainerDied","Data":"58f9322fc4903e3e0844a1a7796430f1607e64b7128dd2663aefde73fea22663"} Nov 03 20:32:14.894648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.894608 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7564-e4c772"] Nov 03 20:32:14.895441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.894869 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7564-e4c772" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" containerName="test-curl-7564-e4c772" containerID="cri-o://aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a" gracePeriod=2 Nov 03 20:32:14.908414 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:14.905621 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:14.919287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:14.919258 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7564-e4c772"] Nov 03 20:32:15.376832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.372564 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6"] Nov 03 20:32:15.376832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.372862 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" containerID="cri-o://71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b" gracePeriod=30 Nov 03 20:32:15.376832 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:15.375864 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:15.643532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.627072 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:32:15.733195 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:15.733154 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:15.877632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.877350 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir\") pod \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " Nov 03 20:32:15.878905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.878880 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache\") pod \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " Nov 03 20:32:15.879586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.879567 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm\") pod \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " Nov 03 20:32:15.880061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.880041 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs\") pod \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " Nov 03 20:32:15.880746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.880725 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home\") pod \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " Nov 03 20:32:15.881248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.881228 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz\" (UniqueName: \"kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz\") pod \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\" (UID: \"279ab75f-c9e2-4d5a-9907-a608f7633b7a\") " Nov 03 20:32:15.881597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.878774 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir" pod "279ab75f-c9e2-4d5a-9907-a608f7633b7a" (UID: "279ab75f-c9e2-4d5a-9907-a608f7633b7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:15.881597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.879437 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache" pod "279ab75f-c9e2-4d5a-9907-a608f7633b7a" (UID: "279ab75f-c9e2-4d5a-9907-a608f7633b7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:15.881597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.881110 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home" pod "279ab75f-c9e2-4d5a-9907-a608f7633b7a" (UID: "279ab75f-c9e2-4d5a-9907-a608f7633b7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:15.882386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.882361 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.882586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.882569 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.882724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.882708 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.898875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.898806 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" event={"ID":"279ab75f-c9e2-4d5a-9907-a608f7633b7a","Type":"ContainerDied","Data":"4fb75b0edcfb7a00be640fa044a9822237ea2002478a9fcefc33ef5191bad0b8"} Nov 03 20:32:15.899075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.899056 5815 scope.go:122] "RemoveContainer" containerID="58f9322fc4903e3e0844a1a7796430f1607e64b7128dd2663aefde73fea22663" Nov 03 20:32:15.899341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.899324 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" Nov 03 20:32:15.924171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.924140 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs" pod "279ab75f-c9e2-4d5a-9907-a608f7633b7a" (UID: "279ab75f-c9e2-4d5a-9907-a608f7633b7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:15.924383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.924174 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm" pod "279ab75f-c9e2-4d5a-9907-a608f7633b7a" (UID: "279ab75f-c9e2-4d5a-9907-a608f7633b7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:15.924950 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:15.924918 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:15.926589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.926548 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz" pod "279ab75f-c9e2-4d5a-9907-a608f7633b7a" (UID: "279ab75f-c9e2-4d5a-9907-a608f7633b7a"). InnerVolumeSpecName "kube-api-access-hlsbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:15.990548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.988360 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlsbz\" (UniqueName: \"kubernetes.io/projected/279ab75f-c9e2-4d5a-9907-a608f7633b7a-kube-api-access-hlsbz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.990548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.988388 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279ab75f-c9e2-4d5a-9907-a608f7633b7a-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.990548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:15.988397 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279ab75f-c9e2-4d5a-9907-a608f7633b7a-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:16.219320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.218883 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-j9gl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:16.219320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.218946 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:16.252847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.252818 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5"] Nov 03 20:32:16.260076 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:16.260049 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:16.262404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.262386 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5"] Nov 03 20:32:16.277218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.277183 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/auth-test-model-ba89e0-kserve-57877fc457-8xnm5" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:16.367616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.367580 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf"] Nov 03 20:32:16.368689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368667 5815 cpu_manager.go:414] "containerMap: removing container" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" Nov 03 20:32:16.368790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368691 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" Nov 03 20:32:16.368790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368736 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" containerName="test-curl-7564-e4c772" Nov 03 20:32:16.368790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368746 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" containerName="test-curl-7564-e4c772" Nov 03 20:32:16.368790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368770 5815 cpu_manager.go:414] "containerMap: removing container" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" Nov 03 20:32:16.368790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368778 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" Nov 03 20:32:16.368991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368936 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="744520ef-cd92-40a9-a060-c97a0af7c33e" containerName="main" Nov 03 20:32:16.368991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368951 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" containerName="test-curl-7564-e4c772" Nov 03 20:32:16.368991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.368964 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" containerName="main" Nov 03 20:32:16.374683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.374661 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.382925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.381681 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ed5ad5\"/\"openshift-service-ca.crt\"" Nov 03 20:32:16.382925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.381822 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ed5ad5\"/\"default-dockercfg-qdgxm\"" Nov 03 20:32:16.382925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.381885 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ed5ad5\"/\"kube-root-ca.crt\"" Nov 03 20:32:16.382925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.382441 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ed5ad5\"/\"noten-model-ed5ad5-kserve-self-signed-certs\"" Nov 03 20:32:16.390095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.388304 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf"] Nov 03 20:32:16.402685 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:16.402398 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:16.413977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.413559 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.413977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.413607 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.413977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.413665 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.413977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.413890 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.414206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.414041 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7k64\" (UniqueName: \"kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.414206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.414093 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515111 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7k64\" (UniqueName: \"kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515178 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515237 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515257 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515534 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515785 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.515947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515924 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.516044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.515955 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.517922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.517900 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.518299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.518278 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.528355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.528324 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7k64\" (UniqueName: \"kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64\") pod \"noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.670529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.661424 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279ab75f-c9e2-4d5a-9907-a608f7633b7a" path="/var/lib/kubelet/pods/279ab75f-c9e2-4d5a-9907-a608f7633b7a/volumes" Nov 03 20:32:16.718530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.717476 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:16.867458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.865753 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerName="main" containerID="cri-o://3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8" gracePeriod=45 Nov 03 20:32:16.876073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:16.876032 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.220:8000/ready\": read tcp 10.132.0.2:50258->10.132.0.220:8000: read: connection reset by peer" Nov 03 20:32:17.165445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.164940 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf"] Nov 03 20:32:17.169557 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:17.169495 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a7998dc_f6e0_4817_8694_b3146004be67.slice/crio-b8e34c7c2bacc441714b20a41f2a709d952e273406541a058f1176f65a15fea2 WatchSource:0}: Error finding container b8e34c7c2bacc441714b20a41f2a709d952e273406541a058f1176f65a15fea2: Status 404 returned error can't find the container with id b8e34c7c2bacc441714b20a41f2a709d952e273406541a058f1176f65a15fea2 Nov 03 20:32:17.193647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.193618 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:17.240882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.240858 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:32:17.340530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.330085 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk\" (UniqueName: \"kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk\") pod \"ba71efa8-290a-4a65-a976-bb9fae54ca16\" (UID: \"ba71efa8-290a-4a65-a976-bb9fae54ca16\") " Nov 03 20:32:17.362532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.359773 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk" pod "ba71efa8-290a-4a65-a976-bb9fae54ca16" (UID: "ba71efa8-290a-4a65-a976-bb9fae54ca16"). InnerVolumeSpecName "kube-api-access-8qskk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.473081 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm\") pod \"70d684d2-cade-4c87-a6c7-2f138c02272b\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.473314 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf\" (UniqueName: \"kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf\") pod \"70d684d2-cade-4c87-a6c7-2f138c02272b\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.473364 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs\" (UniqueName: \"kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs\") pod \"70d684d2-cade-4c87-a6c7-2f138c02272b\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.473629 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache\") pod \"70d684d2-cade-4c87-a6c7-2f138c02272b\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.473666 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home\") pod \"70d684d2-cade-4c87-a6c7-2f138c02272b\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.473713 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir\") pod \"70d684d2-cade-4c87-a6c7-2f138c02272b\" (UID: \"70d684d2-cade-4c87-a6c7-2f138c02272b\") " Nov 03 20:32:17.485259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.474371 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qskk\" (UniqueName: \"kubernetes.io/projected/ba71efa8-290a-4a65-a976-bb9fae54ca16-kube-api-access-8qskk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.509631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.495406 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home" pod "70d684d2-cade-4c87-a6c7-2f138c02272b" (UID: "70d684d2-cade-4c87-a6c7-2f138c02272b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:17.509631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.507660 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache" pod "70d684d2-cade-4c87-a6c7-2f138c02272b" (UID: "70d684d2-cade-4c87-a6c7-2f138c02272b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:17.521854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.514461 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir" pod "70d684d2-cade-4c87-a6c7-2f138c02272b" (UID: "70d684d2-cade-4c87-a6c7-2f138c02272b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:17.605677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.590648 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm" pod "70d684d2-cade-4c87-a6c7-2f138c02272b" (UID: "70d684d2-cade-4c87-a6c7-2f138c02272b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:17.605677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.591781 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.605677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.591807 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.605677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.591822 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.605677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.591834 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d684d2-cade-4c87-a6c7-2f138c02272b-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.711348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.711223 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" containerID="cri-o://407ad0341f37759e98b15adff63d2a89d531a88dcf1edceb0c52d653ac54dc22" gracePeriod=45 Nov 03 20:32:17.787694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.787617 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf" pod "70d684d2-cade-4c87-a6c7-2f138c02272b" (UID: "70d684d2-cade-4c87-a6c7-2f138c02272b"). InnerVolumeSpecName "kube-api-access-bpvwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:17.794127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.794075 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs" pod "70d684d2-cade-4c87-a6c7-2f138c02272b" (UID: "70d684d2-cade-4c87-a6c7-2f138c02272b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:17.824469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.824410 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpvwf\" (UniqueName: \"kubernetes.io/projected/70d684d2-cade-4c87-a6c7-2f138c02272b-kube-api-access-bpvwf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.824469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.824444 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d684d2-cade-4c87-a6c7-2f138c02272b-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.926648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.926614 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" event={"ID":"0a7998dc-f6e0-4817-8694-b3146004be67","Type":"ContainerStarted","Data":"e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9"} Nov 03 20:32:17.926807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.926661 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" event={"ID":"0a7998dc-f6e0-4817-8694-b3146004be67","Type":"ContainerStarted","Data":"b8e34c7c2bacc441714b20a41f2a709d952e273406541a058f1176f65a15fea2"} Nov 03 20:32:17.928687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.928650 5815 generic.go:356] "Generic (PLEG): container finished" podID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerID="407ad0341f37759e98b15adff63d2a89d531a88dcf1edceb0c52d653ac54dc22" exitCode=0 Nov 03 20:32:17.928825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.928726 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" event={"ID":"1449fe1f-bbae-48c3-a531-4dea8c03c441","Type":"ContainerDied","Data":"407ad0341f37759e98b15adff63d2a89d531a88dcf1edceb0c52d653ac54dc22"} Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.930591 5815 generic.go:356] "Generic (PLEG): container finished" podID="ba71efa8-290a-4a65-a976-bb9fae54ca16" containerID="aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a" exitCode=137 Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.930659 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.930733 5815 scope.go:122] "RemoveContainer" containerID="aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a" Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.933170 5815 generic.go:356] "Generic (PLEG): container finished" podID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerID="3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8" exitCode=0 Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.933217 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" event={"ID":"70d684d2-cade-4c87-a6c7-2f138c02272b","Type":"ContainerDied","Data":"3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8"} Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.933244 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" event={"ID":"70d684d2-cade-4c87-a6c7-2f138c02272b","Type":"ContainerDied","Data":"70b26ef8d01815e1939f1f340616ce3a04e6f587886460530d5a3c74b703f1b1"} Nov 03 20:32:17.934194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.933301 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc" Nov 03 20:32:17.949628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.949577 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" podStartSLOduration=1.949559173 podStartE2EDuration="1.949559173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:17.948903054 +0000 UTC m=+1847.479248728" watchObservedRunningTime="2026-11-03 20:32:17.949559173 +0000 UTC m=+1847.479904847" Nov 03 20:32:17.962535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.959642 5815 scope.go:122] "RemoveContainer" containerID="aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a" Nov 03 20:32:17.962535 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:17.960913 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a\": container with ID starting with aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a not found: ID does not exist" containerID="aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a" Nov 03 20:32:17.962535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.960970 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a"} err="failed to get container status \"aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a\": rpc error: code = NotFound desc = could not find container \"aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a\": container with ID starting with aa09162c348856173f916ae5735f12f61aa72ca1562eba442ad27263d8e60d1a not found: ID does not exist" Nov 03 20:32:17.962535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.960995 5815 scope.go:122] "RemoveContainer" containerID="3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8" Nov 03 20:32:17.987721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.978041 5815 scope.go:122] "RemoveContainer" containerID="3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8" Nov 03 20:32:17.987721 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:17.979780 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8\": container with ID starting with 3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8 not found: ID does not exist" containerID="3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8" Nov 03 20:32:17.987721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.979815 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8"} err="failed to get container status \"3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8\": rpc error: code = NotFound desc = could not find container \"3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8\": container with ID starting with 3c6a8e00a6d1b6aca267b66d6b54117f1dedf626cbf136eba9781947c1205cc8 not found: ID does not exist" Nov 03 20:32:17.987721 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:17.983807 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:17.987721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.985922 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc"] Nov 03 20:32:17.991158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.991118 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-30e9de/auth-test-model-30e9de-kserve-5dd47c8965-slwkc"] Nov 03 20:32:17.999623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:17.999284 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": dial tcp 10.132.0.223:8000: connect: connection refused" Nov 03 20:32:18.257611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.257584 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:32:18.314538 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:18.308745 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7564-e4c772\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" pod="opendatahub/test-curl-7564-e4c772" Nov 03 20:32:18.532087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.531997 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home\") pod \"1449fe1f-bbae-48c3-a531-4dea8c03c441\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " Nov 03 20:32:18.532087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532067 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache\") pod \"1449fe1f-bbae-48c3-a531-4dea8c03c441\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532112 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir\") pod \"1449fe1f-bbae-48c3-a531-4dea8c03c441\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532155 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs\" (UniqueName: \"kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs\") pod \"1449fe1f-bbae-48c3-a531-4dea8c03c441\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532240 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp\" (UniqueName: \"kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp\") pod \"1449fe1f-bbae-48c3-a531-4dea8c03c441\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532348 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home" pod "1449fe1f-bbae-48c3-a531-4dea8c03c441" (UID: "1449fe1f-bbae-48c3-a531-4dea8c03c441"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532378 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache" pod "1449fe1f-bbae-48c3-a531-4dea8c03c441" (UID: "1449fe1f-bbae-48c3-a531-4dea8c03c441"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532435 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm\") pod \"1449fe1f-bbae-48c3-a531-4dea8c03c441\" (UID: \"1449fe1f-bbae-48c3-a531-4dea8c03c441\") " Nov 03 20:32:18.532659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.532554 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir" pod "1449fe1f-bbae-48c3-a531-4dea8c03c441" (UID: "1449fe1f-bbae-48c3-a531-4dea8c03c441"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:18.533290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.533268 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:18.533381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.533295 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:18.533381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.533308 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:18.535661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.535635 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm" pod "1449fe1f-bbae-48c3-a531-4dea8c03c441" (UID: "1449fe1f-bbae-48c3-a531-4dea8c03c441"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:18.535806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.535678 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs" pod "1449fe1f-bbae-48c3-a531-4dea8c03c441" (UID: "1449fe1f-bbae-48c3-a531-4dea8c03c441"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:18.535874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.535834 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp" pod "1449fe1f-bbae-48c3-a531-4dea8c03c441" (UID: "1449fe1f-bbae-48c3-a531-4dea8c03c441"). InnerVolumeSpecName "kube-api-access-nbxkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:18.588887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.588853 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" path="/var/lib/kubelet/pods/70d684d2-cade-4c87-a6c7-2f138c02272b/volumes" Nov 03 20:32:18.589362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.589347 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba71efa8-290a-4a65-a976-bb9fae54ca16" path="/var/lib/kubelet/pods/ba71efa8-290a-4a65-a976-bb9fae54ca16/volumes" Nov 03 20:32:18.635082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.635058 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1449fe1f-bbae-48c3-a531-4dea8c03c441-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:18.635082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.635083 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1449fe1f-bbae-48c3-a531-4dea8c03c441-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:18.635254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.635093 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxkp\" (UniqueName: \"kubernetes.io/projected/1449fe1f-bbae-48c3-a531-4dea8c03c441-kube-api-access-nbxkp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:18.717929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.717903 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:18.948226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.948133 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" Nov 03 20:32:18.948408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.948142 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j" event={"ID":"1449fe1f-bbae-48c3-a531-4dea8c03c441","Type":"ContainerDied","Data":"6c4793d60fcab367f428b9cfe61d77395202be5d6f5f083fe15ec3f8b6195acc"} Nov 03 20:32:18.948408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.948271 5815 scope.go:122] "RemoveContainer" containerID="407ad0341f37759e98b15adff63d2a89d531a88dcf1edceb0c52d653ac54dc22" Nov 03 20:32:18.971675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.971648 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j"] Nov 03 20:32:18.979187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:18.979157 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/auth-test-model-6e5139-kserve-745bfdfb9c-ndq6j"] Nov 03 20:32:20.591781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:20.591744 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" path="/var/lib/kubelet/pods/1449fe1f-bbae-48c3-a531-4dea8c03c441/volumes" Nov 03 20:32:20.992165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:20.992138 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:32:21.030949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.030909 5815 generic.go:356] "Generic (PLEG): container finished" podID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerID="71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b" exitCode=0 Nov 03 20:32:21.031282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.031226 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" event={"ID":"369568fe-f4a1-4dd5-904a-eb30dae71013","Type":"ContainerDied","Data":"71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b"} Nov 03 20:32:21.031443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.031285 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" event={"ID":"369568fe-f4a1-4dd5-904a-eb30dae71013","Type":"ContainerDied","Data":"17b3db4a69913fa3ad56cdb4be00a8d0ef5e4c349afb8fefa0ec3b39f6f6d4e2"} Nov 03 20:32:21.031443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.031319 5815 scope.go:122] "RemoveContainer" containerID="71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b" Nov 03 20:32:21.031657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.031452 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6" Nov 03 20:32:21.050384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.050363 5815 scope.go:122] "RemoveContainer" containerID="b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347" Nov 03 20:32:21.062296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.062277 5815 scope.go:122] "RemoveContainer" containerID="71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b" Nov 03 20:32:21.062907 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:21.062881 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b\": container with ID starting with 71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b not found: ID does not exist" containerID="71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b" Nov 03 20:32:21.062997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.062915 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b"} err="failed to get container status \"71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b\": rpc error: code = NotFound desc = could not find container \"71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b\": container with ID starting with 71f4a5bcd928449ae7ccaf00d119dda1bc9280a5973c5788b7a5b4ebd7fdf41b not found: ID does not exist" Nov 03 20:32:21.062997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.062934 5815 scope.go:122] "RemoveContainer" containerID="b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347" Nov 03 20:32:21.063241 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:21.063221 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347\": container with ID starting with b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347 not found: ID does not exist" containerID="b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347" Nov 03 20:32:21.063295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.063246 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347"} err="failed to get container status \"b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347\": rpc error: code = NotFound desc = could not find container \"b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347\": container with ID starting with b44a1e56d8a6c349a58200fa8c5e88befb465410feb57c4824836651a8eaf347 not found: ID does not exist" Nov 03 20:32:21.113118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113082 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.113296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113131 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.113296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113177 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.113296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113249 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.113843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113611 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:21.113843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113742 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.113843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.113831 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.114092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114015 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114161 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114196 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114280 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo\") pod \"369568fe-f4a1-4dd5-904a-eb30dae71013\" (UID: \"369568fe-f4a1-4dd5-904a-eb30dae71013\") " Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114549 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114721 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.114814 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.115380 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/369568fe-f4a1-4dd5-904a-eb30dae71013-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.115534 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.115598 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.115614 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.115662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.115629 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.116744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.116718 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.116991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.116943 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:21.117074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.117033 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "kube-api-access-pdjcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:21.117216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.117201 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo" pod "369568fe-f4a1-4dd5-904a-eb30dae71013" (UID: "369568fe-f4a1-4dd5-904a-eb30dae71013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:21.218773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.218165 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.218773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.218189 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjcr\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-kube-api-access-pdjcr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.218773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.218203 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.218773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.218211 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/369568fe-f4a1-4dd5-904a-eb30dae71013-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.373337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.373246 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6"] Nov 03 20:32:21.379949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:21.379921 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-j9gl6"] Nov 03 20:32:22.404684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.404649 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp"] Nov 03 20:32:22.405876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405817 5815 cpu_manager.go:414] "containerMap: removing container" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" Nov 03 20:32:22.405876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405839 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" Nov 03 20:32:22.406049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405894 5815 cpu_manager.go:414] "containerMap: removing container" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" Nov 03 20:32:22.406049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405902 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" Nov 03 20:32:22.406049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405949 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" Nov 03 20:32:22.406049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405959 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" Nov 03 20:32:22.406049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405973 5815 cpu_manager.go:414] "containerMap: removing container" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerName="main" Nov 03 20:32:22.406049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.405980 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerName="main" Nov 03 20:32:22.406355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.406131 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1449fe1f-bbae-48c3-a531-4dea8c03c441" containerName="main" Nov 03 20:32:22.406355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.406145 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d684d2-cade-4c87-a6c7-2f138c02272b" containerName="main" Nov 03 20:32:22.406355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.406157 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" Nov 03 20:32:22.424166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.424135 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.438528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.433920 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-db5198\"" Nov 03 20:32:22.479820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.479782 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp"] Nov 03 20:32:22.486711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.486686 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g"] Nov 03 20:32:22.487957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.487930 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" containerName="istio-proxy" Nov 03 20:32:22.493807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.493787 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g"] Nov 03 20:32:22.493942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.493920 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:22.500264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.499028 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-db5198-dockercfg-zz6bl\"" Nov 03 20:32:22.514828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.514801 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tmxl\" (UniqueName: \"kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl\") pod \"payload-processing-e2e-extmod-a-db5198-666565cf58-659tp\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.516036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.516015 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-db5198-666565cf58-659tp\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.632534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.623173 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tmxl\" (UniqueName: \"kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl\") pod \"payload-processing-e2e-extmod-a-db5198-666565cf58-659tp\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.632842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.632815 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-db5198-666565cf58-659tp\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.641664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.633959 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-db5198-666565cf58-659tp\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.683586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.682232 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tmxl\" (UniqueName: \"kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl\") pod \"payload-processing-e2e-extmod-a-db5198-666565cf58-659tp\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.687528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.686407 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369568fe-f4a1-4dd5-904a-eb30dae71013" path="/var/lib/kubelet/pods/369568fe-f4a1-4dd5-904a-eb30dae71013/volumes" Nov 03 20:32:22.736793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.736755 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997vp\" (UniqueName: \"kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp\") pod \"payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:22.737847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.737818 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:22.762816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.762780 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:22.845538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.842693 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:22.845538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.842961 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997vp\" (UniqueName: \"kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp\") pod \"payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:22.881539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.868279 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:22.922724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:22.922647 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997vp\" (UniqueName: \"kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp\") pod \"payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:23.129149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:23.127886 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:23.558613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:23.558578 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf"] Nov 03 20:32:23.751154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:23.737234 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv"] Nov 03 20:32:23.751154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:23.737493 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" podUID="4e264e86-9138-4e30-8dc1-90aee8ba7746" containerName="istio-proxy" containerID="cri-o://dadd6f4c9c1b14e716ae008803c10576d51ec1d030de756b544ec2ea49d384b4" gracePeriod=30 Nov 03 20:32:24.767978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:24.763633 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:25.087730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:25.087698 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp"] Nov 03 20:32:25.409532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:25.403392 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" containerID="cri-o://1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9" gracePeriod=45 Nov 03 20:32:25.516596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:25.515860 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g"] Nov 03 20:32:25.525813 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:25.525497 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode514b625_a309_4e35_b49f_7fbdadf82c0b.slice/crio-14ce122dea1adbb046635c8707d82753724acfd85cc21e6e3db9e4d224c1e0ae WatchSource:0}: Error finding container 14ce122dea1adbb046635c8707d82753724acfd85cc21e6e3db9e4d224c1e0ae: Status 404 returned error can't find the container with id 14ce122dea1adbb046635c8707d82753724acfd85cc21e6e3db9e4d224c1e0ae Nov 03 20:32:25.587907 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:25.587829 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc916591_d0b9_48e3_a940_e605f410ccb5.slice/crio-1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc916591_d0b9_48e3_a940_e605f410ccb5.slice/crio-conmon-1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:32:25.593218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:25.593181 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" event={"ID":"69967678-8d12-435a-8d46-341b7485f73e","Type":"ContainerStarted","Data":"1e0f0c0b95b6ed9dfa78143ec90e912aa5f9adece3bd145918cbd415ea8d7451"} Nov 03 20:32:25.647367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:25.647329 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/ready\": dial tcp 10.132.0.221:8000: connect: connection refused" Nov 03 20:32:25.824902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:25.824879 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:32:26.040714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.040677 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs\") pod \"bc916591-d0b9-48e3-a940-e605f410ccb5\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " Nov 03 20:32:26.040946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.040800 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir\") pod \"bc916591-d0b9-48e3-a940-e605f410ccb5\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " Nov 03 20:32:26.041065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.041045 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x\" (UniqueName: \"kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x\") pod \"bc916591-d0b9-48e3-a940-e605f410ccb5\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " Nov 03 20:32:26.041148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.041091 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache\") pod \"bc916591-d0b9-48e3-a940-e605f410ccb5\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " Nov 03 20:32:26.041148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.041111 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm\") pod \"bc916591-d0b9-48e3-a940-e605f410ccb5\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " Nov 03 20:32:26.041285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.041196 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home\") pod \"bc916591-d0b9-48e3-a940-e605f410ccb5\" (UID: \"bc916591-d0b9-48e3-a940-e605f410ccb5\") " Nov 03 20:32:26.042028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.042005 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home" pod "bc916591-d0b9-48e3-a940-e605f410ccb5" (UID: "bc916591-d0b9-48e3-a940-e605f410ccb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.042679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.042608 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir" pod "bc916591-d0b9-48e3-a940-e605f410ccb5" (UID: "bc916591-d0b9-48e3-a940-e605f410ccb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.042787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.042740 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache" pod "bc916591-d0b9-48e3-a940-e605f410ccb5" (UID: "bc916591-d0b9-48e3-a940-e605f410ccb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.044435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.044414 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs" pod "bc916591-d0b9-48e3-a940-e605f410ccb5" (UID: "bc916591-d0b9-48e3-a940-e605f410ccb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:26.044814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.044795 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x" pod "bc916591-d0b9-48e3-a940-e605f410ccb5" (UID: "bc916591-d0b9-48e3-a940-e605f410ccb5"). InnerVolumeSpecName "kube-api-access-hks7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:26.045041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.045022 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm" pod "bc916591-d0b9-48e3-a940-e605f410ccb5" (UID: "bc916591-d0b9-48e3-a940-e605f410ccb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.163535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.160140 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hks7x\" (UniqueName: \"kubernetes.io/projected/bc916591-d0b9-48e3-a940-e605f410ccb5-kube-api-access-hks7x\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.163535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.160182 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.163535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.160212 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.163535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.160226 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.163535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.160238 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc916591-d0b9-48e3-a940-e605f410ccb5-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.163535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.160251 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc916591-d0b9-48e3-a940-e605f410ccb5-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.607426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.607394 5815 generic.go:356] "Generic (PLEG): container finished" podID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerID="1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9" exitCode=0 Nov 03 20:32:26.607618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.607431 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" Nov 03 20:32:26.607618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.607474 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" event={"ID":"bc916591-d0b9-48e3-a940-e605f410ccb5","Type":"ContainerDied","Data":"1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9"} Nov 03 20:32:26.607618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.607538 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8" event={"ID":"bc916591-d0b9-48e3-a940-e605f410ccb5","Type":"ContainerDied","Data":"2a54f43f1097e0f66639c42b4e747e7b50d70dec1643b36d2ac52dcb749d0bf9"} Nov 03 20:32:26.607618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.607561 5815 scope.go:122] "RemoveContainer" containerID="1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9" Nov 03 20:32:26.611993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.611888 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" event={"ID":"e514b625-a309-4e35-b49f-7fbdadf82c0b","Type":"ContainerStarted","Data":"64fa0082267425f332563ef26109b6d3538de732cc5de856d1c3bd85d705d11e"} Nov 03 20:32:26.611993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.611936 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:26.611993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.611958 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" event={"ID":"e514b625-a309-4e35-b49f-7fbdadf82c0b","Type":"ContainerStarted","Data":"14ce122dea1adbb046635c8707d82753724acfd85cc21e6e3db9e4d224c1e0ae"} Nov 03 20:32:26.613606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.613581 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" event={"ID":"69967678-8d12-435a-8d46-341b7485f73e","Type":"ContainerStarted","Data":"b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9"} Nov 03 20:32:26.613729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.613661 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:26.624611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.624573 5815 scope.go:122] "RemoveContainer" containerID="1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9" Nov 03 20:32:26.627096 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:26.627069 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9\": container with ID starting with 1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9 not found: ID does not exist" containerID="1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9" Nov 03 20:32:26.627204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.627125 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9"} err="failed to get container status \"1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9\": rpc error: code = NotFound desc = could not find container \"1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9\": container with ID starting with 1d558d77211f0f5098df4d076862ef27e030e94a02b2a83f6a2474c28eb6f1b9 not found: ID does not exist" Nov 03 20:32:26.632299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.632260 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" podStartSLOduration=4.632248417 podStartE2EDuration="4.632248417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:26.630364658 +0000 UTC m=+1856.160710331" watchObservedRunningTime="2026-11-03 20:32:26.632248417 +0000 UTC m=+1856.162594088" Nov 03 20:32:26.650631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.650595 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" podStartSLOduration=4.650583971 podStartE2EDuration="4.650583971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:26.648465466 +0000 UTC m=+1856.178811140" watchObservedRunningTime="2026-11-03 20:32:26.650583971 +0000 UTC m=+1856.180929643" Nov 03 20:32:26.666246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.666222 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8"] Nov 03 20:32:26.673036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:26.673015 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/auth-test-model-99a8fe-kserve-7b998b64c7-2qbp8"] Nov 03 20:32:28.589530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:28.589486 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" path="/var/lib/kubelet/pods/bc916591-d0b9-48e3-a940-e605f410ccb5/volumes" Nov 03 20:32:29.643602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:29.643571 5815 generic.go:356] "Generic (PLEG): container finished" podID="4e264e86-9138-4e30-8dc1-90aee8ba7746" containerID="dadd6f4c9c1b14e716ae008803c10576d51ec1d030de756b544ec2ea49d384b4" exitCode=0 Nov 03 20:32:29.644023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:29.643645 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" event={"ID":"4e264e86-9138-4e30-8dc1-90aee8ba7746","Type":"ContainerDied","Data":"dadd6f4c9c1b14e716ae008803c10576d51ec1d030de756b544ec2ea49d384b4"} Nov 03 20:32:30.108646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.108607 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx"] Nov 03 20:32:30.109996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.109775 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" containerID="cri-o://1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4" gracePeriod=30 Nov 03 20:32:30.118381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.118354 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5"] Nov 03 20:32:30.118779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.118752 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" containerID="cri-o://6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139" gracePeriod=30 Nov 03 20:32:30.148142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.148118 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:30.269156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269119 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.269549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269411 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.269787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269761 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.269958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269883 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.269958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269929 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.270149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269961 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.270149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.269992 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.270149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.270039 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.270371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.270214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs\") pod \"4e264e86-9138-4e30-8dc1-90aee8ba7746\" (UID: \"4e264e86-9138-4e30-8dc1-90aee8ba7746\") " Nov 03 20:32:30.271810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.271250 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:30.271958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.271812 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:30.272743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.272721 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:30.272905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.272887 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:30.273287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.273265 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:30.275423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.275389 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:30.281243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.281220 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:30.281356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.281341 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:30.282957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.282937 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449" pod "4e264e86-9138-4e30-8dc1-90aee8ba7746" (UID: "4e264e86-9138-4e30-8dc1-90aee8ba7746"). InnerVolumeSpecName "kube-api-access-lw449". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384295 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384334 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384350 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw449\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-kube-api-access-lw449\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384362 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384373 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384388 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e264e86-9138-4e30-8dc1-90aee8ba7746-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384400 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384411 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e264e86-9138-4e30-8dc1-90aee8ba7746-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.387027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.384423 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e264e86-9138-4e30-8dc1-90aee8ba7746-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:30.654572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.654535 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" event={"ID":"4e264e86-9138-4e30-8dc1-90aee8ba7746","Type":"ContainerDied","Data":"4921ecb76b724aad704c8c0f550b03449f7d47e59607d28d9a00467fc536fffe"} Nov 03 20:32:30.654572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.654568 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv" Nov 03 20:32:30.655076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.654605 5815 scope.go:122] "RemoveContainer" containerID="dadd6f4c9c1b14e716ae008803c10576d51ec1d030de756b544ec2ea49d384b4" Nov 03 20:32:30.679098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.676967 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv"] Nov 03 20:32:30.684390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:30.684335 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ed5ad5-openshift-default-85499958df-jmwvv"] Nov 03 20:32:32.204412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.204375 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl"] Nov 03 20:32:32.205538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.205492 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4e264e86-9138-4e30-8dc1-90aee8ba7746" containerName="istio-proxy" Nov 03 20:32:32.205741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.205714 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e264e86-9138-4e30-8dc1-90aee8ba7746" containerName="istio-proxy" Nov 03 20:32:32.205868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.205752 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" Nov 03 20:32:32.205868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.205761 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" Nov 03 20:32:32.206018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.205993 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e264e86-9138-4e30-8dc1-90aee8ba7746" containerName="istio-proxy" Nov 03 20:32:32.206087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.206022 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc916591-d0b9-48e3-a940-e605f410ccb5" containerName="main" Nov 03 20:32:32.212123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.212105 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.237550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.237499 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-05473b-openshift-default-dockercfg-25bff\"" Nov 03 20:32:32.249746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.249697 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl"] Nov 03 20:32:32.333968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.333921 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.333988 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334119 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334158 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334302 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334336 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334498 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7tf\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334535 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.334763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.334608 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.439538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.438101 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7tf\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.439538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.438613 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.439538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.439409 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.439890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.439869 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.440256 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.440600 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.440737 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.441026 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.441096 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.441140 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.441285 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.441327 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.446024 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.448531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.447139 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.464534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.462085 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.486530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.471680 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.501530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.499818 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.551531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.544161 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7tf\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.572531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.570845 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:32.668085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:32.668049 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e264e86-9138-4e30-8dc1-90aee8ba7746" path="/var/lib/kubelet/pods/4e264e86-9138-4e30-8dc1-90aee8ba7746/volumes" Nov 03 20:32:33.098632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.098553 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl"] Nov 03 20:32:33.114062 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:33.110342 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dfcfbf_8392_4b60_a96a_7427ee3ad9d1.slice/crio-7eb662f3064977de9dd0bc95a80d6bfd9359a475755841c1041172207c098a56 WatchSource:0}: Error finding container 7eb662f3064977de9dd0bc95a80d6bfd9359a475755841c1041172207c098a56: Status 404 returned error can't find the container with id 7eb662f3064977de9dd0bc95a80d6bfd9359a475755841c1041172207c098a56 Nov 03 20:32:33.140961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.134404 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:33.140961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.134530 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:33.140961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.134581 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:33.739593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.738773 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" event={"ID":"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1","Type":"ContainerStarted","Data":"ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798"} Nov 03 20:32:33.739593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.738839 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" event={"ID":"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1","Type":"ContainerStarted","Data":"7eb662f3064977de9dd0bc95a80d6bfd9359a475755841c1041172207c098a56"} Nov 03 20:32:33.770213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:33.770149 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podStartSLOduration=1.7701316459999998 podStartE2EDuration="1.770131646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:33.76305853 +0000 UTC m=+1863.293404202" watchObservedRunningTime="2026-11-03 20:32:33.770131646 +0000 UTC m=+1863.300477318" Nov 03 20:32:34.572085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:34.572050 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:34.586604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:34.586578 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:34.748281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:34.748250 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:34.749222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:34.749202 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:32:35.474083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.474023 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:32:35.503012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.502987 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:32:35.569689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.569654 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.569876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.569705 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.569876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.569742 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.569876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.569803 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.569876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.569837 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.570182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.569935 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.570297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.570380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.570380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570057 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.570380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570078 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.570380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570181 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.570380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570322 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:35.570380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570273 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.570734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570427 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.570814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570793 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:35.570905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570887 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.570996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.570933 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.571062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571008 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.571120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571060 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.571120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571114 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.571236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571138 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.571362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571344 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.571437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571422 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.571569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571552 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy\") pod \"805307b8-5902-4bc3-8d13-002a1cc438c6\" (UID: \"805307b8-5902-4bc3-8d13-002a1cc438c6\") " Nov 03 20:32:35.571648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571570 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.571648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571575 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.571648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.571591 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket\") pod \"bc257930-c536-4240-9e51-712cb9721b71\" (UID: \"bc257930-c536-4240-9e51-712cb9721b71\") " Nov 03 20:32:35.572583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572498 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.572922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572898 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805307b8-5902-4bc3-8d13-002a1cc438c6-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572929 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572943 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572955 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572982 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.572995 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.573007 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.573022 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.573035 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.573382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.573063 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc257930-c536-4240-9e51-712cb9721b71-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.582114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582057 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.582114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582063 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:35.582114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582080 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.582340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582107 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.582340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582131 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "kube-api-access-q7ct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.582340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582276 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "kube-api-access-xdcrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:35.582566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.582380 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo" pod "bc257930-c536-4240-9e51-712cb9721b71" (UID: "bc257930-c536-4240-9e51-712cb9721b71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:35.583440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.583408 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy" pod "805307b8-5902-4bc3-8d13-002a1cc438c6" (UID: "805307b8-5902-4bc3-8d13-002a1cc438c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676524 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7ct6\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-kube-api-access-q7ct6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676569 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdcrs\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-kube-api-access-xdcrs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676587 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc257930-c536-4240-9e51-712cb9721b71-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676603 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc257930-c536-4240-9e51-712cb9721b71-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676618 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676634 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676650 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805307b8-5902-4bc3-8d13-002a1cc438c6-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.705586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.676667 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc257930-c536-4240-9e51-712cb9721b71-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:35.771900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.771840 5815 generic.go:356] "Generic (PLEG): container finished" podID="bc257930-c536-4240-9e51-712cb9721b71" containerID="6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139" exitCode=0 Nov 03 20:32:35.772429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.771898 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" event={"ID":"bc257930-c536-4240-9e51-712cb9721b71","Type":"ContainerDied","Data":"6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139"} Nov 03 20:32:35.772429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.771938 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" Nov 03 20:32:35.772429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.771980 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5" event={"ID":"bc257930-c536-4240-9e51-712cb9721b71","Type":"ContainerDied","Data":"1ac604942b32cf0492a623d04b48a7150b37089aeee9cfb58bdff5dcae59cc67"} Nov 03 20:32:35.772429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.772007 5815 scope.go:122] "RemoveContainer" containerID="6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139" Nov 03 20:32:35.775287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.775257 5815 generic.go:356] "Generic (PLEG): container finished" podID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerID="1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4" exitCode=0 Nov 03 20:32:35.775658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.775642 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" Nov 03 20:32:35.775758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.775667 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" event={"ID":"805307b8-5902-4bc3-8d13-002a1cc438c6","Type":"ContainerDied","Data":"1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4"} Nov 03 20:32:35.775758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.775720 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx" event={"ID":"805307b8-5902-4bc3-8d13-002a1cc438c6","Type":"ContainerDied","Data":"0cee4ecf61eed5a6d2ff5540a99f28e0f235f55c5cd4b71e240b57ba1649ecc5"} Nov 03 20:32:35.793316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.793199 5815 scope.go:122] "RemoveContainer" containerID="6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139" Nov 03 20:32:35.793634 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:35.793613 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139\": container with ID starting with 6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139 not found: ID does not exist" containerID="6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139" Nov 03 20:32:35.793797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.793660 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139"} err="failed to get container status \"6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139\": rpc error: code = NotFound desc = could not find container \"6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139\": container with ID starting with 6bf83bbb9c996a85272c6ace4a791511808f9cc8dba10e91ed0137f3101da139 not found: ID does not exist" Nov 03 20:32:35.793797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.793686 5815 scope.go:122] "RemoveContainer" containerID="1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4" Nov 03 20:32:35.807815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.807791 5815 scope.go:122] "RemoveContainer" containerID="1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4" Nov 03 20:32:35.808196 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:35.808169 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4\": container with ID starting with 1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4 not found: ID does not exist" containerID="1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4" Nov 03 20:32:35.808283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.808206 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4"} err="failed to get container status \"1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4\": rpc error: code = NotFound desc = could not find container \"1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4\": container with ID starting with 1774dc6b2b2f9b3906140a01de12d6a90debd59f600d7efdb8e539e63f0a7fd4 not found: ID does not exist" Nov 03 20:32:35.810072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.810048 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5"] Nov 03 20:32:35.816262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.816239 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dhjv5"] Nov 03 20:32:35.838184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.838154 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx"] Nov 03 20:32:35.850064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:35.850037 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-dv4kx"] Nov 03 20:32:36.009441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:36.009397 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x"] Nov 03 20:32:36.009657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:36.009630 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" containerID="cri-o://f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d" gracePeriod=30 Nov 03 20:32:36.588301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:36.588269 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" path="/var/lib/kubelet/pods/805307b8-5902-4bc3-8d13-002a1cc438c6/volumes" Nov 03 20:32:36.588870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:36.588854 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc257930-c536-4240-9e51-712cb9721b71" path="/var/lib/kubelet/pods/bc257930-c536-4240-9e51-712cb9721b71/volumes" Nov 03 20:32:37.258020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.257985 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh"] Nov 03 20:32:37.259066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.259046 5815 cpu_manager.go:414] "containerMap: removing container" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" Nov 03 20:32:37.259399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.259365 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" Nov 03 20:32:37.259483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.259417 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" Nov 03 20:32:37.259483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.259426 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" Nov 03 20:32:37.259690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.259676 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="805307b8-5902-4bc3-8d13-002a1cc438c6" containerName="istio-proxy" Nov 03 20:32:37.259757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.259694 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc257930-c536-4240-9e51-712cb9721b71" containerName="istio-proxy" Nov 03 20:32:37.265651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.265621 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.271286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.271256 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh"] Nov 03 20:32:37.281818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.281796 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-05473b-dockercfg-kphct\"" Nov 03 20:32:37.286719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.286695 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-05473b\"" Nov 03 20:32:37.295530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.295312 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t"] Nov 03 20:32:37.315863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.313923 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.316836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.316810 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t"] Nov 03 20:32:37.323539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.323441 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.323539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.323487 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldkl\" (UniqueName: \"kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl\") pod \"payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.443532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.441950 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smcpc\" (UniqueName: \"kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc\") pod \"payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.443532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.442237 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.443532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.442276 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldkl\" (UniqueName: \"kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl\") pod \"payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.443532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.442343 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.443532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.443393 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.548579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.545072 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.548579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.545267 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smcpc\" (UniqueName: \"kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc\") pod \"payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.548579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.546360 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.548579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.547163 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldkl\" (UniqueName: \"kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl\") pod \"payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.603543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.594167 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smcpc\" (UniqueName: \"kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc\") pod \"payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.621536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.620499 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:37.650227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.646991 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 20:32:37.650227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.648218 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:37.650227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:37.648604 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 20:32:38.408535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:38.407964 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t"] Nov 03 20:32:38.497367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:38.497341 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh"] Nov 03 20:32:38.505581 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:38.505553 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bacd210_dadc_449f_8f13_228fe7cecc3c.slice/crio-12f2b6ad58c9529fd475ef012d38b9b281e438366c2993e28176c26a2421c554 WatchSource:0}: Error finding container 12f2b6ad58c9529fd475ef012d38b9b281e438366c2993e28176c26a2421c554: Status 404 returned error can't find the container with id 12f2b6ad58c9529fd475ef012d38b9b281e438366c2993e28176c26a2421c554 Nov 03 20:32:38.644940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:38.644896 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" podUID="0a7998dc-f6e0-4817-8694-b3146004be67" containerName="main" containerID="cri-o://e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9" gracePeriod=45 Nov 03 20:32:39.010219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.010197 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.030975 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" event={"ID":"3955adb0-0d7a-4707-90c5-194b5b0e9793","Type":"ContainerStarted","Data":"9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4"} Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.031060 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" event={"ID":"3955adb0-0d7a-4707-90c5-194b5b0e9793","Type":"ContainerStarted","Data":"3347594651d37401dab2542e58c4f4a9123989940dbcd59c0b955c9b4378a534"} Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.032007 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.034206 5815 generic.go:356] "Generic (PLEG): container finished" podID="0a7998dc-f6e0-4817-8694-b3146004be67" containerID="e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9" exitCode=0 Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.034242 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.034303 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" event={"ID":"0a7998dc-f6e0-4817-8694-b3146004be67","Type":"ContainerDied","Data":"e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9"} Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.034366 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf" event={"ID":"0a7998dc-f6e0-4817-8694-b3146004be67","Type":"ContainerDied","Data":"b8e34c7c2bacc441714b20a41f2a709d952e273406541a058f1176f65a15fea2"} Nov 03 20:32:39.038579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.034403 5815 scope.go:122] "RemoveContainer" containerID="e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9" Nov 03 20:32:39.039583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.039556 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" event={"ID":"5bacd210-dadc-449f-8f13-228fe7cecc3c","Type":"ContainerStarted","Data":"81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07"} Nov 03 20:32:39.039657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.039623 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:39.039719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.039655 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" event={"ID":"5bacd210-dadc-449f-8f13-228fe7cecc3c","Type":"ContainerStarted","Data":"12f2b6ad58c9529fd475ef012d38b9b281e438366c2993e28176c26a2421c554"} Nov 03 20:32:39.048661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.048640 5815 scope.go:122] "RemoveContainer" containerID="e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9" Nov 03 20:32:39.048975 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:39.048953 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9\": container with ID starting with e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9 not found: ID does not exist" containerID="e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9" Nov 03 20:32:39.049043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.048980 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9"} err="failed to get container status \"e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9\": rpc error: code = NotFound desc = could not find container \"e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9\": container with ID starting with e8e529947f03f8b8294949dddeb773cc30eb7412bc962d54d83051e83034e7a9 not found: ID does not exist" Nov 03 20:32:39.056288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.056237 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" podStartSLOduration=2.056221404 podStartE2EDuration="2.056221404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:39.052461211 +0000 UTC m=+1868.582806890" watchObservedRunningTime="2026-11-03 20:32:39.056221404 +0000 UTC m=+1868.586567077" Nov 03 20:32:39.057602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.057569 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs\") pod \"0a7998dc-f6e0-4817-8694-b3146004be67\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " Nov 03 20:32:39.057602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.057617 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home\") pod \"0a7998dc-f6e0-4817-8694-b3146004be67\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " Nov 03 20:32:39.057779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.057655 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir\") pod \"0a7998dc-f6e0-4817-8694-b3146004be67\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " Nov 03 20:32:39.057779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.057708 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64\" (UniqueName: \"kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64\") pod \"0a7998dc-f6e0-4817-8694-b3146004be67\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " Nov 03 20:32:39.057896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.057791 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm\") pod \"0a7998dc-f6e0-4817-8694-b3146004be67\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " Nov 03 20:32:39.057963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.057926 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home" pod "0a7998dc-f6e0-4817-8694-b3146004be67" (UID: "0a7998dc-f6e0-4817-8694-b3146004be67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:39.058057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.058034 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache\") pod \"0a7998dc-f6e0-4817-8694-b3146004be67\" (UID: \"0a7998dc-f6e0-4817-8694-b3146004be67\") " Nov 03 20:32:39.058562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.058272 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir" pod "0a7998dc-f6e0-4817-8694-b3146004be67" (UID: "0a7998dc-f6e0-4817-8694-b3146004be67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:39.058562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.058411 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache" pod "0a7998dc-f6e0-4817-8694-b3146004be67" (UID: "0a7998dc-f6e0-4817-8694-b3146004be67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:39.059056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.058833 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.059056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.058901 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.059056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.059031 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.062654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.062634 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm" pod "0a7998dc-f6e0-4817-8694-b3146004be67" (UID: "0a7998dc-f6e0-4817-8694-b3146004be67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:39.062654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.062646 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs" pod "0a7998dc-f6e0-4817-8694-b3146004be67" (UID: "0a7998dc-f6e0-4817-8694-b3146004be67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:39.062787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.062675 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64" pod "0a7998dc-f6e0-4817-8694-b3146004be67" (UID: "0a7998dc-f6e0-4817-8694-b3146004be67"). InnerVolumeSpecName "kube-api-access-f7k64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:39.080574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.080501 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" podStartSLOduration=2.080483416 podStartE2EDuration="2.080483416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:39.077053352 +0000 UTC m=+1868.607399016" watchObservedRunningTime="2026-11-03 20:32:39.080483416 +0000 UTC m=+1868.610829097" Nov 03 20:32:39.160344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.160314 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a7998dc-f6e0-4817-8694-b3146004be67-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.160344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.160344 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7998dc-f6e0-4817-8694-b3146004be67-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.160567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.160357 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7k64\" (UniqueName: \"kubernetes.io/projected/0a7998dc-f6e0-4817-8694-b3146004be67-kube-api-access-f7k64\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.364291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.364262 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf"] Nov 03 20:32:39.369361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:39.369337 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ed5ad5/noten-model-ed5ad5-kserve-5f6d6f9649-z6gjf"] Nov 03 20:32:40.587055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:40.587020 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7998dc-f6e0-4817-8694-b3146004be67" path="/var/lib/kubelet/pods/0a7998dc-f6e0-4817-8694-b3146004be67/volumes" Nov 03 20:32:41.319561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.319526 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:32:41.394862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.394830 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.394886 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.394916 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395037 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395136 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395265 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395333 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395394 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.395477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395458 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token\") pod \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\" (UID: \"1bcc9bc8-8d66-4ef4-bf80-20619126fced\") " Nov 03 20:32:41.396286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.395985 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:41.396286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.396132 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.396710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.396553 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:41.396989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.396831 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:41.397147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.397118 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:41.397356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.397331 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:41.398889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.398844 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:41.399156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.399136 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:41.407007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.406981 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "kube-api-access-dszfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:41.411673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.411647 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo" pod "1bcc9bc8-8d66-4ef4-bf80-20619126fced" (UID: "1bcc9bc8-8d66-4ef4-bf80-20619126fced"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:41.498440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498392 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498443 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498459 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498472 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498487 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dszfv\" (UniqueName: \"kubernetes.io/projected/1bcc9bc8-8d66-4ef4-bf80-20619126fced-kube-api-access-dszfv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498521 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498536 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bcc9bc8-8d66-4ef4-bf80-20619126fced-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.498724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.498550 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bcc9bc8-8d66-4ef4-bf80-20619126fced-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:41.675585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.675041 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj"] Nov 03 20:32:41.695529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.690369 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0a7998dc-f6e0-4817-8694-b3146004be67" containerName="main" Nov 03 20:32:41.695529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.690398 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7998dc-f6e0-4817-8694-b3146004be67" containerName="main" Nov 03 20:32:41.695529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.690458 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" Nov 03 20:32:41.695529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.690468 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" Nov 03 20:32:41.695529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.691171 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7998dc-f6e0-4817-8694-b3146004be67" containerName="main" Nov 03 20:32:41.695529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.691198 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerName="istio-proxy" Nov 03 20:32:41.722705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.722675 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj"] Nov 03 20:32:41.723079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.723021 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:41.732193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.729838 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0f6e09\"/\"kube-root-ca.crt\"" Nov 03 20:32:41.732193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.730571 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0f6e09\"/\"openshift-service-ca.crt\"" Nov 03 20:32:41.802529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.801873 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0f6e09\"/\"explicit-model-0f6e09-kserve-self-signed-certs\"" Nov 03 20:32:41.802529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.802070 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0f6e09\"/\"default-dockercfg-l7jlz\"" Nov 03 20:32:41.888533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.827362 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:41.912534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.888956 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:41.912904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.912872 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:41.913056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.913038 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:41.913283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.913265 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl4sz\" (UniqueName: \"kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:41.928580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:41.913492 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.153740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.132141 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl4sz\" (UniqueName: \"kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.153740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.132263 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.153740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.132338 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.153740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.132457 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.153740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.132545 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.153740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.132568 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.182447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.175840 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.182447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.177176 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.182447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.177383 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.182765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.182718 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.305264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.301939 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl4sz\" (UniqueName: \"kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.357051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.357015 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs\") pod \"explicit-model-0f6e09-kserve-7b495c488c-t8qrj\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.443851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.443767 5815 generic.go:356] "Generic (PLEG): container finished" podID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" containerID="f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d" exitCode=0 Nov 03 20:32:42.444326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.444298 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" event={"ID":"1bcc9bc8-8d66-4ef4-bf80-20619126fced","Type":"ContainerDied","Data":"f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d"} Nov 03 20:32:42.444606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.444581 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" event={"ID":"1bcc9bc8-8d66-4ef4-bf80-20619126fced","Type":"ContainerDied","Data":"982d012b2c11c008ecd8876675b8eba984ee709b805cedda844aee34a90b36b4"} Nov 03 20:32:42.444759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.444745 5815 scope.go:122] "RemoveContainer" containerID="f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d" Nov 03 20:32:42.445021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.445004 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x" Nov 03 20:32:42.453267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.453243 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:42.492086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.492065 5815 scope.go:122] "RemoveContainer" containerID="f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d" Nov 03 20:32:42.492530 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:42.492491 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d\": container with ID starting with f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d not found: ID does not exist" containerID="f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d" Nov 03 20:32:42.492632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.492547 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d"} err="failed to get container status \"f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d\": rpc error: code = NotFound desc = could not find container \"f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d\": container with ID starting with f6ca914a45ac12ac80b17ce1a0f28761fb0bdad131374817bdbd27d6d744848d not found: ID does not exist" Nov 03 20:32:42.506596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.506571 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x"] Nov 03 20:32:42.513007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.512981 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-96w8x"] Nov 03 20:32:42.589677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.589646 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcc9bc8-8d66-4ef4-bf80-20619126fced" path="/var/lib/kubelet/pods/1bcc9bc8-8d66-4ef4-bf80-20619126fced/volumes" Nov 03 20:32:42.743843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.743816 5815 scope.go:122] "RemoveContainer" containerID="ba439f359fba4eaa8195a68096c45b54f83d7eca4079e32d9d02f31d06bdb5d4" Nov 03 20:32:42.761494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.761462 5815 scope.go:122] "RemoveContainer" containerID="d7ec305164ad933b663dab19d1f6a6eed267ba9c98db655cdb3a6debe83b7222" Nov 03 20:32:42.775917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.775897 5815 scope.go:122] "RemoveContainer" containerID="60c07de504fc37e73300da72f93baed9062433be048acef5a5479b42d87a6681" Nov 03 20:32:42.777914 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:42.777883 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod031eb390_3746_43d7_9076_ce5e45a15828.slice/crio-a6c041ba29537ae270a16d0f8826bee2ff6d96b0ffc969701ae0a33c24cddf02 WatchSource:0}: Error finding container a6c041ba29537ae270a16d0f8826bee2ff6d96b0ffc969701ae0a33c24cddf02: Status 404 returned error can't find the container with id a6c041ba29537ae270a16d0f8826bee2ff6d96b0ffc969701ae0a33c24cddf02 Nov 03 20:32:42.781372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.780158 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj"] Nov 03 20:32:42.791051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.791024 5815 scope.go:122] "RemoveContainer" containerID="22fb07553895321973c0acd8327c1405d087f2cef6bacc8d1e7e93e91bce91f3" Nov 03 20:32:42.807583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.807492 5815 scope.go:122] "RemoveContainer" containerID="14574acc1fecf1f8e886b2417e0b68a05db8872dce1ea37b854822d9edd4e1db" Nov 03 20:32:42.819857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.819835 5815 scope.go:122] "RemoveContainer" containerID="2e09a00474448f4b6fc6a22946256aef20b195bbf8801a8800f082986bed9a52" Nov 03 20:32:42.832390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:42.832373 5815 scope.go:122] "RemoveContainer" containerID="6e6231bc09714ff5e91a54f7dbab5302d09ccd5f62f25c793ce90a4491d9abe3" Nov 03 20:32:43.459743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:43.459649 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" event={"ID":"031eb390-3746-43d7-9076-ce5e45a15828","Type":"ContainerStarted","Data":"ea92dbe23413ef729b21d78847c6054c0eccd2aabe53ccaf3121be89b503b0e9"} Nov 03 20:32:43.459743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:43.459699 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" event={"ID":"031eb390-3746-43d7-9076-ce5e45a15828","Type":"ContainerStarted","Data":"a6c041ba29537ae270a16d0f8826bee2ff6d96b0ffc969701ae0a33c24cddf02"} Nov 03 20:32:43.481202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:43.481142 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" podStartSLOduration=2.481123892 podStartE2EDuration="2.481123892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:43.478809629 +0000 UTC m=+1873.009155302" watchObservedRunningTime="2026-11-03 20:32:43.481123892 +0000 UTC m=+1873.011469578" Nov 03 20:32:43.854309 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:32:43.854269 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:32:44.453966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:44.453929 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:32:45.874312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:45.874280 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl"] Nov 03 20:32:45.874736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:45.874478 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" containerID="cri-o://017a2bff9c8abe1eaa05856f4fd549dab0346f7a76f5b7e33ca54b4480e0d69e" gracePeriod=30 Nov 03 20:32:50.083416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:50.080551 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 20:32:50.086143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:50.086116 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 20:32:51.029555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.029521 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj"] Nov 03 20:32:51.653037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.653000 5815 generic.go:356] "Generic (PLEG): container finished" podID="6e7fb429-124e-4008-894f-fc3262dc9682" containerID="017a2bff9c8abe1eaa05856f4fd549dab0346f7a76f5b7e33ca54b4480e0d69e" exitCode=0 Nov 03 20:32:51.653496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.653239 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" event={"ID":"6e7fb429-124e-4008-894f-fc3262dc9682","Type":"ContainerDied","Data":"017a2bff9c8abe1eaa05856f4fd549dab0346f7a76f5b7e33ca54b4480e0d69e"} Nov 03 20:32:51.678054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.678025 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:32:51.764034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.763998 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.766648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.766617 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770077 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770186 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770248 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770289 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770381 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770433 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.773538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.770643 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p\") pod \"6e7fb429-124e-4008-894f-fc3262dc9682\" (UID: \"6e7fb429-124e-4008-894f-fc3262dc9682\") " Nov 03 20:32:51.810869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.802750 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:51.810869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.803007 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:51.810869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.803218 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:51.810869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.808643 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:51.810869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.809584 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:51.831839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.831758 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:51.880595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.874733 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "kube-api-access-97g4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:51.935630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.934019 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:51.935630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.934176 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token" pod "6e7fb429-124e-4008-894f-fc3262dc9682" (UID: "6e7fb429-124e-4008-894f-fc3262dc9682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950406 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950449 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950464 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97g4p\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-kube-api-access-97g4p\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950480 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950494 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e7fb429-124e-4008-894f-fc3262dc9682-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950526 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e7fb429-124e-4008-894f-fc3262dc9682-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950601 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e7fb429-124e-4008-894f-fc3262dc9682-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950621 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:51.956531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:51.950634 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e7fb429-124e-4008-894f-fc3262dc9682-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:52.662822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.662791 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" Nov 03 20:32:52.663273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.662827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl" event={"ID":"6e7fb429-124e-4008-894f-fc3262dc9682","Type":"ContainerDied","Data":"8b264f5b7f6ea7a3af5c0071b2b8cc06afad88c40250d1ad362ec2ad525e2316"} Nov 03 20:32:52.663273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.662894 5815 scope.go:122] "RemoveContainer" containerID="017a2bff9c8abe1eaa05856f4fd549dab0346f7a76f5b7e33ca54b4480e0d69e" Nov 03 20:32:52.687936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.687907 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl"] Nov 03 20:32:52.693390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.693356 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-vjgvl"] Nov 03 20:32:52.926482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.926414 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q"] Nov 03 20:32:52.927274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.927249 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" Nov 03 20:32:52.927274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.927275 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" Nov 03 20:32:52.927491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.927458 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" containerName="istio-proxy" Nov 03 20:32:52.937239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.934662 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:52.950729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.946395 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9fbc61\"/\"kube-root-ca.crt\"" Nov 03 20:32:52.950729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.946413 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-9fbc61\"/\"openshift-service-ca.crt\"" Nov 03 20:32:52.950729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.946400 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9fbc61\"/\"autores-model-9fbc61-kserve-self-signed-certs\"" Nov 03 20:32:52.950729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.946454 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-9fbc61\"/\"default-dockercfg-xp74b\"" Nov 03 20:32:52.950729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.948952 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q"] Nov 03 20:32:52.975573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.974841 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:52.975573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.975066 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:52.975573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.975103 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:52.975573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.975243 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:52.975573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.975278 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76mz\" (UniqueName: \"kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:52.975573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:52.975315 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.081176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.081139 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.081864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.081742 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.081864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.081864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.082117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.082004 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.082117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.082041 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76mz\" (UniqueName: \"kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.082117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.082102 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.082281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.082203 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.082751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.082427 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.082751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.082524 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.120560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.110160 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.120560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.114457 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76mz\" (UniqueName: \"kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.121617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.121426 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm\") pod \"autores-model-9fbc61-kserve-54457f65cc-qmf4q\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.265547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.263783 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:53.968759 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:53.967711 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92558514_de20_4c62_91d8_5bd86ce37761.slice/crio-c87ae96ccbd8595ebb9d34ad3416deff286e3d539b3e4716b79f13912f8e14dd WatchSource:0}: Error finding container c87ae96ccbd8595ebb9d34ad3416deff286e3d539b3e4716b79f13912f8e14dd: Status 404 returned error can't find the container with id c87ae96ccbd8595ebb9d34ad3416deff286e3d539b3e4716b79f13912f8e14dd Nov 03 20:32:53.969232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:53.969159 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q"] Nov 03 20:32:54.644544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:54.638461 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7fb429-124e-4008-894f-fc3262dc9682" path="/var/lib/kubelet/pods/6e7fb429-124e-4008-894f-fc3262dc9682/volumes" Nov 03 20:32:54.861222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:54.859345 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" event={"ID":"92558514-de20-4c62-91d8-5bd86ce37761","Type":"ContainerStarted","Data":"ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea"} Nov 03 20:32:54.861222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:54.859400 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" event={"ID":"92558514-de20-4c62-91d8-5bd86ce37761","Type":"ContainerStarted","Data":"c87ae96ccbd8595ebb9d34ad3416deff286e3d539b3e4716b79f13912f8e14dd"} Nov 03 20:32:55.286533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:55.277792 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:32:58.454411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.454322 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" podStartSLOduration=6.454303858 podStartE2EDuration="6.454303858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:54.982554611 +0000 UTC m=+1884.512900350" watchObservedRunningTime="2026-11-03 20:32:58.454303858 +0000 UTC m=+1887.984649531" Nov 03 20:32:58.456857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.456825 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7"] Nov 03 20:32:58.462333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.462313 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.506580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.503859 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-421604-openshift-default-dockercfg-p242f\"" Nov 03 20:32:58.515404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.512482 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7"] Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.644706 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.644806 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.644923 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.644945 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.644985 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.645032 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.645066 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.645138 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwd8z\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.645532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.645184 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798188 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798320 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798495 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798633 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798673 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798896 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.798957 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.799057 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwd8z\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.799540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.799138 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.803532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.800799 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.803532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.801435 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.803532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.801951 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.803532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.802353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.819877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.819845 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.909531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.886987 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.952262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.952180 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:58.953159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:58.953104 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwd8z\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:59.047927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:59.047832 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:59.126481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:59.126394 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:32:59.507088 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:32:59.507057 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346a0e2d_1843_4125_bc93_b6bf25993c3b.slice/crio-f6ea7eb5ec3616a6f0981eff82e36fe4acea6432152c6f8109273a3569a7f32a WatchSource:0}: Error finding container f6ea7eb5ec3616a6f0981eff82e36fe4acea6432152c6f8109273a3569a7f32a: Status 404 returned error can't find the container with id f6ea7eb5ec3616a6f0981eff82e36fe4acea6432152c6f8109273a3569a7f32a Nov 03 20:32:59.510041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:59.510012 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7"] Nov 03 20:32:59.511084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:59.511041 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:59.511196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:59.511131 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:32:59.511196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:32:59.511173 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:00.088565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:00.088522 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" event={"ID":"346a0e2d-1843-4125-bc93-b6bf25993c3b","Type":"ContainerStarted","Data":"1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3"} Nov 03 20:33:00.088763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:00.088584 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" event={"ID":"346a0e2d-1843-4125-bc93-b6bf25993c3b","Type":"ContainerStarted","Data":"f6ea7eb5ec3616a6f0981eff82e36fe4acea6432152c6f8109273a3569a7f32a"} Nov 03 20:33:00.121996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:00.121947 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podStartSLOduration=2.121931333 podStartE2EDuration="2.121931333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:00.117294614 +0000 UTC m=+1889.647640287" watchObservedRunningTime="2026-11-03 20:33:00.121931333 +0000 UTC m=+1889.652277005" Nov 03 20:33:00.127408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:00.127374 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:33:00.133654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:00.133631 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:33:01.097569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:01.097496 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:33:01.098715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:01.098695 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:33:03.265323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.265286 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:33:03.279569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.279540 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:33:03.465100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.465067 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9"] Nov 03 20:33:03.470185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.470164 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.479785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.479723 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.479974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.479805 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5j58\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.479974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.479894 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.479974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.479955 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.480568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.479981 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.480568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.480029 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.480568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.480180 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.480568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.480224 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.480568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.480271 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.480896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.480791 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9fbc61-openshift-default-dockercfg-v2stb\"" Nov 03 20:33:03.525817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.525685 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9"] Nov 03 20:33:03.531002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.530978 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz"] Nov 03 20:33:03.536805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.536785 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz"] Nov 03 20:33:03.536968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.536898 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.581472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581428 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.581710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581485 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.581710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581532 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.581710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581570 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.581710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581618 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.581710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581661 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.581710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581689 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581746 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581773 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5j58\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581821 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581856 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581904 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581917 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.581957 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvkks\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.582010 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.582039 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.582079 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.582672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.582219 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.582672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.582255 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.582672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.582491 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.583453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.583425 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.583631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.583501 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.584345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.584325 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.592032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.592008 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.592181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.592111 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.617557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.617487 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.620967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.620772 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5j58\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.684164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684127 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvkks\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684342 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684384 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684477 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684558 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684640 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684771 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.684884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684876 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.685004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684951 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.685004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684969 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.685004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.684986 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.685469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.685441 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.686181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.685833 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.687688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.687638 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.689131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.689107 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.697456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.697428 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.698080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.698064 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvkks\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-n46jz\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:03.726384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.726362 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r"] Nov 03 20:33:03.732636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.732618 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:03.739039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.739015 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-421604\"" Nov 03 20:33:03.739187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.739168 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-421604-dockercfg-4ktsz\"" Nov 03 20:33:03.739885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.739862 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r"] Nov 03 20:33:03.786660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.786562 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmp8\" (UniqueName: \"kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8\") pod \"payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:03.787332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.787193 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:03.791962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.791943 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj"] Nov 03 20:33:03.796904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.796613 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:03.800240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.800033 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj"] Nov 03 20:33:03.800240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.800132 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:03.864540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:03.854847 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:04.048591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.021691 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6pk4\" (UniqueName: \"kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4\") pod \"payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.048591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.021773 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.078694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.065957 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:04.165611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.128722 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:04.165611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.128889 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmp8\" (UniqueName: \"kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8\") pod \"payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:04.237175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.230448 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6pk4\" (UniqueName: \"kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4\") pod \"payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.237175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.230548 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.237175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.231889 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.254023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.241869 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmp8\" (UniqueName: \"kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8\") pod \"payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:04.259896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.257373 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6pk4\" (UniqueName: \"kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4\") pod \"payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.291270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.291237 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:33:04.403694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.400783 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:04.554816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.554781 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:04.560772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.559886 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9"] Nov 03 20:33:04.577759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.577697 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz"] Nov 03 20:33:04.585625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.585388 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:04.585625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.585474 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:04.585625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.585555 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:04.586122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.586070 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:04.586227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.586151 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:04.586227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.586190 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:04.823832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.823799 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r"] Nov 03 20:33:04.921583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:04.921034 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj"] Nov 03 20:33:05.270218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.270088 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" event={"ID":"e62cfde2-4364-452b-818c-43450e0ce767","Type":"ContainerStarted","Data":"fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37"} Nov 03 20:33:05.270218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.270153 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" event={"ID":"e62cfde2-4364-452b-818c-43450e0ce767","Type":"ContainerStarted","Data":"5593134c025667399a039ade8e6a75cee98135025ca825cd176d512d1ffc4e1d"} Nov 03 20:33:05.270218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.270187 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:05.273107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.273043 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" event={"ID":"fa2a804a-a656-4bd8-803e-02007913f36d","Type":"ContainerStarted","Data":"ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878"} Nov 03 20:33:05.273275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.273227 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" event={"ID":"fa2a804a-a656-4bd8-803e-02007913f36d","Type":"ContainerStarted","Data":"41d8bdb92d2ee57668b0c2d052ce1ffb0a057dd00ca5c93f80c86f63cf29bcd1"} Nov 03 20:33:05.273275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.273261 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:05.275642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.275574 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" event={"ID":"6c65f969-4693-49e8-8b9e-e8049c21a5cc","Type":"ContainerStarted","Data":"2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f"} Nov 03 20:33:05.275906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.275884 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" event={"ID":"6c65f969-4693-49e8-8b9e-e8049c21a5cc","Type":"ContainerStarted","Data":"a9b191ceb413dcc5a5b3408582661f9ec522fdb0c4778c189f773805f1f23150"} Nov 03 20:33:05.278603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.278581 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" event={"ID":"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305","Type":"ContainerStarted","Data":"c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17"} Nov 03 20:33:05.278720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.278624 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" event={"ID":"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305","Type":"ContainerStarted","Data":"40712fff507a9909565200f927d5961df38199bab8892d9b095be000662c2e8b"} Nov 03 20:33:05.293387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.293333 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" podStartSLOduration=2.29331599 podStartE2EDuration="2.29331599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.288109022 +0000 UTC m=+1894.818454697" watchObservedRunningTime="2026-11-03 20:33:05.29331599 +0000 UTC m=+1894.823661664" Nov 03 20:33:05.322053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.322009 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podStartSLOduration=2.321995669 podStartE2EDuration="2.321995669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.317369992 +0000 UTC m=+1894.847715666" watchObservedRunningTime="2026-11-03 20:33:05.321995669 +0000 UTC m=+1894.852341335" Nov 03 20:33:05.344448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.344398 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podStartSLOduration=2.3443853629999998 podStartE2EDuration="2.344385363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.339555746 +0000 UTC m=+1894.869901422" watchObservedRunningTime="2026-11-03 20:33:05.344385363 +0000 UTC m=+1894.874731035" Nov 03 20:33:05.362949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.362872 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" podStartSLOduration=2.362854734 podStartE2EDuration="2.362854734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.357463871 +0000 UTC m=+1894.887809543" watchObservedRunningTime="2026-11-03 20:33:05.362854734 +0000 UTC m=+1894.893200408" Nov 03 20:33:05.801788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.801742 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:05.807578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.807474 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:05.809184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.809152 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:05.864156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.864130 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:05.868936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.868610 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:05.868936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:05.868672 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:06.088591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.087905 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" podUID="031eb390-3746-43d7-9076-ce5e45a15828" containerName="main" containerID="cri-o://ea92dbe23413ef729b21d78847c6054c0eccd2aabe53ccaf3121be89b503b0e9" gracePeriod=45 Nov 03 20:33:06.327538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.323834 5815 generic.go:356] "Generic (PLEG): container finished" podID="031eb390-3746-43d7-9076-ce5e45a15828" containerID="ea92dbe23413ef729b21d78847c6054c0eccd2aabe53ccaf3121be89b503b0e9" exitCode=0 Nov 03 20:33:06.327538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.325292 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" event={"ID":"031eb390-3746-43d7-9076-ce5e45a15828","Type":"ContainerDied","Data":"ea92dbe23413ef729b21d78847c6054c0eccd2aabe53ccaf3121be89b503b0e9"} Nov 03 20:33:06.602531 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:33:06.601633 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod031eb390_3746_43d7_9076_ce5e45a15828.slice/crio-conmon-ea92dbe23413ef729b21d78847c6054c0eccd2aabe53ccaf3121be89b503b0e9.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:06.851012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.849526 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:06.851012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.849585 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:06.864478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.864394 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:06.864749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:06.864724 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:07.136585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.136055 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:33:07.317648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.317608 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache\") pod \"031eb390-3746-43d7-9076-ce5e45a15828\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " Nov 03 20:33:07.331981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.325643 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs\" (UniqueName: \"kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs\") pod \"031eb390-3746-43d7-9076-ce5e45a15828\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " Nov 03 20:33:07.334902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.334868 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm\") pod \"031eb390-3746-43d7-9076-ce5e45a15828\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " Nov 03 20:33:07.335135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.335119 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir\") pod \"031eb390-3746-43d7-9076-ce5e45a15828\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " Nov 03 20:33:07.335353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.335335 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz\" (UniqueName: \"kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz\") pod \"031eb390-3746-43d7-9076-ce5e45a15828\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " Nov 03 20:33:07.335522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.335489 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home\") pod \"031eb390-3746-43d7-9076-ce5e45a15828\" (UID: \"031eb390-3746-43d7-9076-ce5e45a15828\") " Nov 03 20:33:07.337102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.337079 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home" pod "031eb390-3746-43d7-9076-ce5e45a15828" (UID: "031eb390-3746-43d7-9076-ce5e45a15828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.337470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.337441 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir" pod "031eb390-3746-43d7-9076-ce5e45a15828" (UID: "031eb390-3746-43d7-9076-ce5e45a15828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.339790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.339762 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache" pod "031eb390-3746-43d7-9076-ce5e45a15828" (UID: "031eb390-3746-43d7-9076-ce5e45a15828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.395625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.387942 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz" pod "031eb390-3746-43d7-9076-ce5e45a15828" (UID: "031eb390-3746-43d7-9076-ce5e45a15828"). InnerVolumeSpecName "kube-api-access-kl4sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:07.453045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.452470 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl4sz\" (UniqueName: \"kubernetes.io/projected/031eb390-3746-43d7-9076-ce5e45a15828-kube-api-access-kl4sz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.453045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.452529 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.453045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.452548 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.453045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.452562 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.453045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.452656 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm" pod "031eb390-3746-43d7-9076-ce5e45a15828" (UID: "031eb390-3746-43d7-9076-ce5e45a15828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:07.453457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.453160 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs" pod "031eb390-3746-43d7-9076-ce5e45a15828" (UID: "031eb390-3746-43d7-9076-ce5e45a15828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:07.460756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.460719 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" event={"ID":"031eb390-3746-43d7-9076-ce5e45a15828","Type":"ContainerDied","Data":"a6c041ba29537ae270a16d0f8826bee2ff6d96b0ffc969701ae0a33c24cddf02"} Nov 03 20:33:07.460902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.460770 5815 scope.go:122] "RemoveContainer" containerID="ea92dbe23413ef729b21d78847c6054c0eccd2aabe53ccaf3121be89b503b0e9" Nov 03 20:33:07.461830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.461798 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj" Nov 03 20:33:07.555547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.554810 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031eb390-3746-43d7-9076-ce5e45a15828-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.555547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.554841 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031eb390-3746-43d7-9076-ce5e45a15828-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:07.638530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.637752 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj"] Nov 03 20:33:07.647684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.647601 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0f6e09/explicit-model-0f6e09-kserve-7b495c488c-t8qrj"] Nov 03 20:33:07.815535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.814694 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:07.815535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.814756 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:07.864177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.864143 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:07.864352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:07.864199 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:08.843594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:08.804251 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:08.843594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:08.804310 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:08.952775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:08.952737 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:08.953068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:08.953040 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:09.028640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.028604 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031eb390-3746-43d7-9076-ce5e45a15828" path="/var/lib/kubelet/pods/031eb390-3746-43d7-9076-ce5e45a15828/volumes" Nov 03 20:33:09.029639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.029615 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh"] Nov 03 20:33:09.031110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.031087 5815 cpu_manager.go:414] "containerMap: removing container" podUID="031eb390-3746-43d7-9076-ce5e45a15828" containerName="main" Nov 03 20:33:09.031312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.031297 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031eb390-3746-43d7-9076-ce5e45a15828" containerName="main" Nov 03 20:33:09.045801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.033065 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="031eb390-3746-43d7-9076-ce5e45a15828" containerName="main" Nov 03 20:33:09.090767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.090737 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.107731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.107557 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh"] Nov 03 20:33:09.151851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.151773 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.153618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.152194 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.153877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.153849 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.159529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.154137 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.159921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.159894 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.160081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.160065 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.160189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.160174 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.160397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.160383 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.160567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.160552 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xfl\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.181018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.180984 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d8d0fa-openshift-default-dockercfg-22xx5\"" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.266545 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.266673 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.266810 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.266844 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.266875 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.266960 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.267018 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xfl\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.267072 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.267529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.267132 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.268231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.267642 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.268231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.267914 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.285531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.268561 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.285531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.283653 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.286158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.286131 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.352638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.320083 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.364907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.353545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xfl\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.364907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.354328 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.365383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.365288 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy\") pod \"e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.450543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.448316 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:09.523544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.501690 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:09.825195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.825158 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:09.825457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.825417 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:09.869558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.868897 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:09.869558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:09.868950 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:10.244552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.234366 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh"] Nov 03 20:33:10.348607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.348558 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:10.348881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.348856 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:10.349067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.349044 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:10.534641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.531574 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q"] Nov 03 20:33:10.919265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.919229 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:10.935088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.935041 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:10.996571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.977205 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:10.996571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:10.977263 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:11.338733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:11.338698 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" event={"ID":"0e375507-e16f-4bb4-8187-39e26e4e5348","Type":"ContainerStarted","Data":"24d48d24defd36a1f6b6c3fe2a0f9eea268acedee72c33ccdfed45e631c7c1ba"} Nov 03 20:33:11.806534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:11.806178 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:11.806534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:11.806241 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:11.864323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:11.864288 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:11.864479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:11.864345 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:12.351575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.351538 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" event={"ID":"0e375507-e16f-4bb4-8187-39e26e4e5348","Type":"ContainerStarted","Data":"8a95ae10a92e560c4b9b641179e4f692f55b4cf88a902cc446a36dd28c78aa23"} Nov 03 20:33:12.390623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.390538 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podStartSLOduration=4.390498458 podStartE2EDuration="4.390498458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:12.389950848 +0000 UTC m=+1901.920296522" watchObservedRunningTime="2026-11-03 20:33:12.390498458 +0000 UTC m=+1901.920844130" Nov 03 20:33:12.449617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.449578 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:12.454592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.454564 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:12.801533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.800918 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" start-of-body= Nov 03 20:33:12.801533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.800982 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.7:15021/healthz/ready\": dial tcp 10.132.1.7:15021: connect: connection refused" Nov 03 20:33:12.870533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.864594 5815 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Nov 03 20:33:12.870533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:12.864684 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Nov 03 20:33:13.410321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.410284 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:13.411750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.411726 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:33:13.802728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.802701 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:13.803275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.803254 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:13.804162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.804138 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:33:13.856220 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:33:13.856105 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:33:13.868966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.868936 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:13.869630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.869554 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:13.872278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:13.872256 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:33:16.172309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.172270 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk"] Nov 03 20:33:16.180907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.180884 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.184022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.183591 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk"] Nov 03 20:33:16.185881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.185857 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d8d0fa-dockercfg-jvr7q\"" Nov 03 20:33:16.250338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.250303 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d8d0fa\"" Nov 03 20:33:16.274740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.274705 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx\") pod \"payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.275084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.275066 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.328571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.328548 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 20:33:16.443606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.419011 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.443606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.419177 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx\") pod \"payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.443606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.421309 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.477538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.446325 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 20:33:16.505293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.505256 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx\") pod \"payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:16.505839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:16.505812 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:17.171919 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:33:17.171884 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee02d163_eb9c_40e4_89a9_687da6fde6bf.slice/crio-4fca6da67878911f52b94ae852977f2c7210a9f959651dd8e1813a25f5d27d6e WatchSource:0}: Error finding container 4fca6da67878911f52b94ae852977f2c7210a9f959651dd8e1813a25f5d27d6e: Status 404 returned error can't find the container with id 4fca6da67878911f52b94ae852977f2c7210a9f959651dd8e1813a25f5d27d6e Nov 03 20:33:17.172052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:17.171931 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk"] Nov 03 20:33:17.830277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:17.830244 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" event={"ID":"ee02d163-eb9c-40e4-89a9-687da6fde6bf","Type":"ContainerStarted","Data":"966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6"} Nov 03 20:33:17.830775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:17.830298 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" event={"ID":"ee02d163-eb9c-40e4-89a9-687da6fde6bf","Type":"ContainerStarted","Data":"4fca6da67878911f52b94ae852977f2c7210a9f959651dd8e1813a25f5d27d6e"} Nov 03 20:33:17.830775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:17.830330 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:17.850700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:17.850652 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" podStartSLOduration=1.850639709 podStartE2EDuration="1.850639709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:17.848888205 +0000 UTC m=+1907.379233878" watchObservedRunningTime="2026-11-03 20:33:17.850639709 +0000 UTC m=+1907.380985381" Nov 03 20:33:26.331527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.328342 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" podUID="92558514-de20-4c62-91d8-5bd86ce37761" containerName="main" containerID="cri-o://ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea" gracePeriod=45 Nov 03 20:33:26.646269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.646239 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:33:26.784637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784605 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache\") pod \"92558514-de20-4c62-91d8-5bd86ce37761\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " Nov 03 20:33:26.784916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784672 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir\") pod \"92558514-de20-4c62-91d8-5bd86ce37761\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " Nov 03 20:33:26.784916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784782 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs\" (UniqueName: \"kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs\") pod \"92558514-de20-4c62-91d8-5bd86ce37761\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " Nov 03 20:33:26.784916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784837 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home\") pod \"92558514-de20-4c62-91d8-5bd86ce37761\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " Nov 03 20:33:26.784916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784870 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm\") pod \"92558514-de20-4c62-91d8-5bd86ce37761\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " Nov 03 20:33:26.784916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784879 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache" pod "92558514-de20-4c62-91d8-5bd86ce37761" (UID: "92558514-de20-4c62-91d8-5bd86ce37761"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:26.785211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.784992 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir" pod "92558514-de20-4c62-91d8-5bd86ce37761" (UID: "92558514-de20-4c62-91d8-5bd86ce37761"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:26.785211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.785071 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz\" (UniqueName: \"kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz\") pod \"92558514-de20-4c62-91d8-5bd86ce37761\" (UID: \"92558514-de20-4c62-91d8-5bd86ce37761\") " Nov 03 20:33:26.785211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.785084 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home" pod "92558514-de20-4c62-91d8-5bd86ce37761" (UID: "92558514-de20-4c62-91d8-5bd86ce37761"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:26.786022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.786000 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.786022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.786021 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.786204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.786031 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.787421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.787397 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs" pod "92558514-de20-4c62-91d8-5bd86ce37761" (UID: "92558514-de20-4c62-91d8-5bd86ce37761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:26.787875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.787855 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm" pod "92558514-de20-4c62-91d8-5bd86ce37761" (UID: "92558514-de20-4c62-91d8-5bd86ce37761"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:26.787952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.787901 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz" pod "92558514-de20-4c62-91d8-5bd86ce37761" (UID: "92558514-de20-4c62-91d8-5bd86ce37761"). InnerVolumeSpecName "kube-api-access-q76mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:26.887737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.887661 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92558514-de20-4c62-91d8-5bd86ce37761-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.887737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.887692 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92558514-de20-4c62-91d8-5bd86ce37761-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.887737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:26.887710 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q76mz\" (UniqueName: \"kubernetes.io/projected/92558514-de20-4c62-91d8-5bd86ce37761-kube-api-access-q76mz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:27.042962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.042921 5815 generic.go:356] "Generic (PLEG): container finished" podID="92558514-de20-4c62-91d8-5bd86ce37761" containerID="ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea" exitCode=0 Nov 03 20:33:27.043145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.042970 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" Nov 03 20:33:27.043145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.042999 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" event={"ID":"92558514-de20-4c62-91d8-5bd86ce37761","Type":"ContainerDied","Data":"ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea"} Nov 03 20:33:27.043145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.043047 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q" event={"ID":"92558514-de20-4c62-91d8-5bd86ce37761","Type":"ContainerDied","Data":"c87ae96ccbd8595ebb9d34ad3416deff286e3d539b3e4716b79f13912f8e14dd"} Nov 03 20:33:27.043145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.043065 5815 scope.go:122] "RemoveContainer" containerID="ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea" Nov 03 20:33:27.056641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.056623 5815 scope.go:122] "RemoveContainer" containerID="ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea" Nov 03 20:33:27.056974 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:33:27.056956 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea\": container with ID starting with ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea not found: ID does not exist" containerID="ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea" Nov 03 20:33:27.057050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.056984 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea"} err="failed to get container status \"ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea\": rpc error: code = NotFound desc = could not find container \"ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea\": container with ID starting with ee7c679fc57c488025f7ea25d7dbe58b85d09dff1a4864474dee0fbf80a0e1ea not found: ID does not exist" Nov 03 20:33:27.072101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.072079 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q"] Nov 03 20:33:27.078415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:27.078393 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-9fbc61/autores-model-9fbc61-kserve-54457f65cc-qmf4q"] Nov 03 20:33:28.588824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:28.588783 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92558514-de20-4c62-91d8-5bd86ce37761" path="/var/lib/kubelet/pods/92558514-de20-4c62-91d8-5bd86ce37761/volumes" Nov 03 20:33:28.880235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:28.880146 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:33:29.651315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:29.651285 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:33:31.956536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:31.950760 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:34.007533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:33.999473 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 20:33:34.049538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:34.046627 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:34.078427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:34.077609 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:43.864094 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:33:43.864057 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:33:44.923454 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:33:44.923418 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332403f8_ebc6_4a19_be4f_ea8a58402fd0.slice/crio-f22ca6498220400941f74c0a4842985e1112446625e50d50bcbceab2dcaff954.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:33:45.888532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.882809 5815 generic.go:356] "Generic (PLEG): container finished" podID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerID="f22ca6498220400941f74c0a4842985e1112446625e50d50bcbceab2dcaff954" exitCode=0 Nov 03 20:33:45.889210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.883842 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" event={"ID":"332403f8-ebc6-4a19-be4f-ea8a58402fd0","Type":"ContainerDied","Data":"f22ca6498220400941f74c0a4842985e1112446625e50d50bcbceab2dcaff954"} Nov 03 20:33:45.897537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.896024 5815 scope.go:122] "RemoveContainer" containerID="f22ca6498220400941f74c0a4842985e1112446625e50d50bcbceab2dcaff954" Nov 03 20:33:45.906747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.903379 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:45.906747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.903623 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:45.906747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.904547 5815 generic.go:356] "Generic (PLEG): container finished" podID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerID="ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798" exitCode=0 Nov 03 20:33:45.906747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.904637 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" event={"ID":"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1","Type":"ContainerDied","Data":"ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798"} Nov 03 20:33:45.906747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.905261 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:45.906747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.905279 5815 scope.go:122] "RemoveContainer" containerID="ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798" Nov 03 20:33:45.911646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.910275 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:45.911646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.910407 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:45.911646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:45.910530 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:33:46.291193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.288857 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt"] Nov 03 20:33:46.291193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.289990 5815 cpu_manager.go:414] "containerMap: removing container" podUID="92558514-de20-4c62-91d8-5bd86ce37761" containerName="main" Nov 03 20:33:46.291193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.290010 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92558514-de20-4c62-91d8-5bd86ce37761" containerName="main" Nov 03 20:33:46.291193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.290237 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="92558514-de20-4c62-91d8-5bd86ce37761" containerName="main" Nov 03 20:33:46.296770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.296747 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.305689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.303088 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt"] Nov 03 20:33:46.308270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.308232 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"payload-processing-external-model-plugins-e2e-extmod-b-421604\"" Nov 03 20:33:46.308598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.308580 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"routing-overlay\"" Nov 03 20:33:46.308822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.308789 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"payload-processing-external-model-e2e--4ce1836f-dockercfg-crmfc\"" Nov 03 20:33:46.308936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.308887 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"kube-root-ca.crt\"" Nov 03 20:33:46.309038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.308955 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"openshift-service-ca.crt\"" Nov 03 20:33:46.309239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.309202 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"agc-cred-eea7befa4349\"" Nov 03 20:33:46.447603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.405358 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.447603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.405524 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.447603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.405626 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.447603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.405668 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwlj2\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.575533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.549656 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.575996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.575955 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.576217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.576199 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwlj2\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.576637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.576611 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.591530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.577902 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.617548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.592163 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.617548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.592695 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.643572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.639543 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwlj2\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:46.929763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.929728 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" event={"ID":"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1","Type":"ContainerStarted","Data":"62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951"} Nov 03 20:33:46.933212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.933167 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" event={"ID":"332403f8-ebc6-4a19-be4f-ea8a58402fd0","Type":"ContainerStarted","Data":"4fcb43d2d402b7c3bf362259d7ea47f85642734407ac634ebf376edc393929e3"} Nov 03 20:33:46.935480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:46.935455 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:47.259126 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:33:47.259099 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19c76ca_2f6f_44c5_96e1_b13a50acc512.slice/crio-fed6dbcfb54486769ff60ee86fe4aebe9c4788ea241d033aee803d38bca123e4 WatchSource:0}: Error finding container fed6dbcfb54486769ff60ee86fe4aebe9c4788ea241d033aee803d38bca123e4: Status 404 returned error can't find the container with id fed6dbcfb54486769ff60ee86fe4aebe9c4788ea241d033aee803d38bca123e4 Nov 03 20:33:47.262080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.262050 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt"] Nov 03 20:33:47.462537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.461861 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:33:47.473600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.469631 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.473600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.469693 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:47.574499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.573835 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:33:47.575718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.575618 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:47.575718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.575676 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:47.944636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.944552 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" event={"ID":"f19c76ca-2f6f-44c5-96e1-b13a50acc512","Type":"ContainerStarted","Data":"585a47af37123348438578daad13203f337059b38e696d8c3daf50b21920179b"} Nov 03 20:33:47.944636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.944608 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" event={"ID":"f19c76ca-2f6f-44c5-96e1-b13a50acc512","Type":"ContainerStarted","Data":"fed6dbcfb54486769ff60ee86fe4aebe9c4788ea241d033aee803d38bca123e4"} Nov 03 20:33:47.944976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:47.944958 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:48.021484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:48.021413 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" podStartSLOduration=2.021395558 podStartE2EDuration="2.021395558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:48.016920412 +0000 UTC m=+1937.547266096" watchObservedRunningTime="2026-11-03 20:33:48.021395558 +0000 UTC m=+1937.551741231" Nov 03 20:33:48.459482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:48.459453 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:48.459692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:48.459519 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:48.583531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:48.582944 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:48.583531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:48.583008 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:49.460556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:49.460485 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.460733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:49.460672 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:49.573542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:49.572892 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:49.573542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:49.572966 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:49.751963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:49.748436 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:33:50.483535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:50.480862 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.483535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:50.480924 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:50.576482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:50.576252 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:50.576817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:50.576783 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:51.241540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:51.230849 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:33:51.459922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:51.459880 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.460110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:51.459935 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:51.608754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:51.608661 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:51.609043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:51.609010 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:52.552535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:52.546969 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.552535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:52.547035 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:52.613532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:52.591617 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:52.613532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:52.591717 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:52.996777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:52.996727 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:33:53.110898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.110843 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:53.235538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.231585 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:53.488529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.466065 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.488870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.488827 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:53.489404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.469597 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:33:53.587360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.586888 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:53.587360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:53.586947 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:54.460490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:54.460456 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.460849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:54.460815 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:54.580533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:54.574877 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:54.580533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:54.574939 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:55.459453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:55.459418 5815 patch_prober.go:29] interesting pod/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.459895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:55.459476 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 20:33:55.571875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:55.571830 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 20:33:55.572064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:55.571902 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 20:33:56.465083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.465012 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:33:56.466654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.466627 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:33:56.481376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.481343 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb"] Nov 03 20:33:56.486335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.486311 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.487336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.487315 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6"] Nov 03 20:33:56.492551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.492527 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.496655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.496633 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb"] Nov 03 20:33:56.503955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.503933 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-03c633-dockercfg-kvrdm\"" Nov 03 20:33:56.504452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.504436 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-03c633\"" Nov 03 20:33:56.506842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.506668 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6"] Nov 03 20:33:56.547222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.547189 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbnj\" (UniqueName: \"kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj\") pod \"payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.547480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.547458 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.547664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.547646 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmgn\" (UniqueName: \"kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn\") pod \"payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.547820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.547785 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.595897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.595864 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:33:56.598593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.598567 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 20:33:56.650644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.650612 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.651719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.651677 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmgn\" (UniqueName: \"kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn\") pod \"payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.652466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.652429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.653920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.653889 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbnj\" (UniqueName: \"kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj\") pod \"payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.673672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.666211 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.680166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.680133 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.736618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.734499 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmgn\" (UniqueName: \"kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn\") pod \"payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:56.755532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.749263 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbnj\" (UniqueName: \"kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj\") pod \"payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.828846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.828806 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:56.855167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:56.852554 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:57.616106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:57.614929 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp"] Nov 03 20:33:57.621517 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:33:57.621470 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f35d794_158d_43b4_84b6_98fce5409ab8.slice/crio-0f648ffbaeb5e24dbd9e261da0e0b1ad22bc18fd082b5b979853712728429eca WatchSource:0}: Error finding container 0f648ffbaeb5e24dbd9e261da0e0b1ad22bc18fd082b5b979853712728429eca: Status 404 returned error can't find the container with id 0f648ffbaeb5e24dbd9e261da0e0b1ad22bc18fd082b5b979853712728429eca Nov 03 20:33:57.638001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:57.637975 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb"] Nov 03 20:33:57.662443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:57.662408 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6"] Nov 03 20:33:58.262753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.262708 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" event={"ID":"d2f8860d-3df3-4242-b9d5-bec04b4699aa","Type":"ContainerStarted","Data":"9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b"} Nov 03 20:33:58.262940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.262805 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" event={"ID":"d2f8860d-3df3-4242-b9d5-bec04b4699aa","Type":"ContainerStarted","Data":"431687b25a8024b263ae83f610cf1ed415be74b0db7274693de2f8ad41198ac1"} Nov 03 20:33:58.262993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.262966 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:33:58.264614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.264576 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" event={"ID":"7f35d794-158d-43b4-84b6-98fce5409ab8","Type":"ContainerStarted","Data":"58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752"} Nov 03 20:33:58.264755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.264632 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" event={"ID":"7f35d794-158d-43b4-84b6-98fce5409ab8","Type":"ContainerStarted","Data":"0f648ffbaeb5e24dbd9e261da0e0b1ad22bc18fd082b5b979853712728429eca"} Nov 03 20:33:58.264755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.264675 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" containerID="cri-o://4fcb43d2d402b7c3bf362259d7ea47f85642734407ac634ebf376edc393929e3" gracePeriod=30 Nov 03 20:33:58.289605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.289524 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" podStartSLOduration=2.289498683 podStartE2EDuration="2.289498683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:58.284827308 +0000 UTC m=+1947.815172981" watchObservedRunningTime="2026-11-03 20:33:58.289498683 +0000 UTC m=+1947.819844355" Nov 03 20:33:58.308353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:58.308304 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" podStartSLOduration=2.308289863 podStartE2EDuration="2.308289863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:58.305703462 +0000 UTC m=+1947.836049135" watchObservedRunningTime="2026-11-03 20:33:58.308289863 +0000 UTC m=+1947.838635560" Nov 03 20:33:59.018270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:59.018245 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:33:59.277348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:59.277274 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:33:59.942430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:59.942400 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh"] Nov 03 20:33:59.946003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:59.945981 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:33:59.958628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:59.956028 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-37aa95-openshift-default-dockercfg-6h6k6\"" Nov 03 20:33:59.968523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:33:59.964730 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh"] Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.139916 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140024 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140147 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140259 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140335 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140397 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.140532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140425 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.141408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.140834 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.156527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.153852 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7cc\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.256646 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.256715 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7cc\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.256817 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.256882 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.256978 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.257059 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.257123 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.257172 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.257531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.257197 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.258184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.258086 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.261595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.258347 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.261595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.259800 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.261595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.259808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.261595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.260104 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.317944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.317907 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.320611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.320582 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.349540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.332108 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.349540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.332439 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7cc\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:00.597586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:00.597253 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:01.070107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.070076 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh"] Nov 03 20:34:01.074039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.073996 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:01.074151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.074084 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:01.074151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.074130 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:01.369048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.368957 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" event={"ID":"e2041c14-e13f-4e68-8008-df150d920310","Type":"ContainerStarted","Data":"fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4"} Nov 03 20:34:01.369048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.369019 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" event={"ID":"e2041c14-e13f-4e68-8008-df150d920310","Type":"ContainerStarted","Data":"bf4abce38065a39b3379fd0387d37208de3e07978c48a5a724e680de31963edb"} Nov 03 20:34:01.408033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.407979 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podStartSLOduration=2.407964035 podStartE2EDuration="2.407964035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:01.401361971 +0000 UTC m=+1950.931707644" watchObservedRunningTime="2026-11-03 20:34:01.407964035 +0000 UTC m=+1950.938309706" Nov 03 20:34:01.607532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:01.606674 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:02.605862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:02.605831 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:03.393158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.393122 5815 generic.go:356] "Generic (PLEG): container finished" podID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerID="4fcb43d2d402b7c3bf362259d7ea47f85642734407ac634ebf376edc393929e3" exitCode=0 Nov 03 20:34:03.393319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.393191 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" event={"ID":"332403f8-ebc6-4a19-be4f-ea8a58402fd0","Type":"ContainerDied","Data":"4fcb43d2d402b7c3bf362259d7ea47f85642734407ac634ebf376edc393929e3"} Nov 03 20:34:03.393319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.393269 5815 scope.go:122] "RemoveContainer" containerID="f22ca6498220400941f74c0a4842985e1112446625e50d50bcbceab2dcaff954" Nov 03 20:34:03.393967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.393935 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:03.395707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.395687 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:34:03.661049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.661025 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:34:03.814532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814119 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.814532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814187 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.814532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814230 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.814532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814395 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.814532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814444 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.814532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814482 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.815036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814582 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.815036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814684 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.815036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.814865 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert\") pod \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\" (UID: \"332403f8-ebc6-4a19-be4f-ea8a58402fd0\") " Nov 03 20:34:03.831531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.820961 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:03.831531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.823700 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:03.831531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.823838 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:03.831531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.827719 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:03.840660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.840629 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:03.874530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.865186 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "kube-api-access-kgq9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:03.883576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.876237 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924226 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924268 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924282 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924297 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgq9d\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-kube-api-access-kgq9d\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924316 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924329 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.924343 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/332403f8-ebc6-4a19-be4f-ea8a58402fd0-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:03.954754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:03.954693 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:04.009529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.005810 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token" pod "332403f8-ebc6-4a19-be4f-ea8a58402fd0" (UID: "332403f8-ebc6-4a19-be4f-ea8a58402fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:04.043620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.043548 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:04.043955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.043936 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/332403f8-ebc6-4a19-be4f-ea8a58402fd0-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:04.405997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.405968 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" Nov 03 20:34:04.406186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.406048 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp" event={"ID":"332403f8-ebc6-4a19-be4f-ea8a58402fd0","Type":"ContainerDied","Data":"2b3d9ec893b4690a77ee693edfecbf0f117174ddc90218130fae730a98d6e0c5"} Nov 03 20:34:04.406186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.406111 5815 scope.go:122] "RemoveContainer" containerID="4fcb43d2d402b7c3bf362259d7ea47f85642734407ac634ebf376edc393929e3" Nov 03 20:34:04.437564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.437532 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp"] Nov 03 20:34:04.443681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.443658 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0f6e09-openshift-default-86b9b7746f-wltkp"] Nov 03 20:34:04.589819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.589745 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" path="/var/lib/kubelet/pods/332403f8-ebc6-4a19-be4f-ea8a58402fd0/volumes" Nov 03 20:34:04.789740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.789708 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7"] Nov 03 20:34:04.790887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.790865 5815 cpu_manager.go:414] "containerMap: removing container" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" Nov 03 20:34:04.790990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.790889 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" Nov 03 20:34:04.790990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.790958 5815 cpu_manager.go:414] "containerMap: removing container" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" Nov 03 20:34:04.790990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.790966 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" Nov 03 20:34:04.791153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.791141 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" Nov 03 20:34:04.791189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.791166 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="332403f8-ebc6-4a19-be4f-ea8a58402fd0" containerName="istio-proxy" Nov 03 20:34:04.795728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.795704 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:04.800905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.800862 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-37aa95\"" Nov 03 20:34:04.807311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.804801 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-37aa95-dockercfg-4s257\"" Nov 03 20:34:04.807311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.807228 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7"] Nov 03 20:34:04.821117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.821089 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq"] Nov 03 20:34:04.826315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.826291 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:04.829401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.829376 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq"] Nov 03 20:34:04.975590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.972649 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fzp\" (UniqueName: \"kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp\") pod \"payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:04.975590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.972756 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcw7f\" (UniqueName: \"kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f\") pod \"payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:04.975590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.972802 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:04.975590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:04.972914 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:05.103532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.080834 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fzp\" (UniqueName: \"kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp\") pod \"payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:05.103901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.103870 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcw7f\" (UniqueName: \"kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f\") pod \"payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:05.104073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.104049 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:05.104313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.104297 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:05.117382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.117350 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:05.154536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.123139 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:05.184541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.180572 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fzp\" (UniqueName: \"kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp\") pod \"payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:05.204533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.199568 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcw7f\" (UniqueName: \"kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f\") pod \"payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:05.420704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.420618 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:05.445851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.445811 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:05.719167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:05.718999 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7"] Nov 03 20:34:06.078710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.078679 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq"] Nov 03 20:34:06.430081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.430046 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" event={"ID":"e3057a3a-a231-4340-8752-d6d7b670f007","Type":"ContainerStarted","Data":"ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5"} Nov 03 20:34:06.430288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.430123 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" event={"ID":"e3057a3a-a231-4340-8752-d6d7b670f007","Type":"ContainerStarted","Data":"b67587e53d43545c0f7eaa91601ea455519791a026abef17e65f6c7947eee97f"} Nov 03 20:34:06.430355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.430296 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:06.432066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.432036 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" event={"ID":"959d9d63-751d-4551-a9d0-8b7f50625066","Type":"ContainerStarted","Data":"54987fae2140f60b231ae3ec200c1cf00382f0672104e18342468afc91e65023"} Nov 03 20:34:06.432181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.432092 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:06.432181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.432116 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" event={"ID":"959d9d63-751d-4551-a9d0-8b7f50625066","Type":"ContainerStarted","Data":"44fe0f8bd420ffc6e9cec9a1ad06c3e48a998ad2d0098f88c86b56564597ab92"} Nov 03 20:34:06.473305 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.473253 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" podStartSLOduration=2.473238519 podStartE2EDuration="2.473238519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:06.46616229 +0000 UTC m=+1955.996507986" watchObservedRunningTime="2026-11-03 20:34:06.473238519 +0000 UTC m=+1956.003584191" Nov 03 20:34:06.493345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:06.493283 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" podStartSLOduration=2.493241646 podStartE2EDuration="2.493241646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:06.485480208 +0000 UTC m=+1956.015825882" watchObservedRunningTime="2026-11-03 20:34:06.493241646 +0000 UTC m=+1956.023587318" Nov 03 20:34:09.295529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:09.290676 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:34:10.398749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:10.398716 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:34:14.000779 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:34:14.000731 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:34:17.139834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.139715 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:34:17.140787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.140434 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:34:17.331168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.331123 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9"] Nov 03 20:34:17.332775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.331367 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" containerID="cri-o://c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17" gracePeriod=30 Nov 03 20:34:17.340103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.340075 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz"] Nov 03 20:34:17.340329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.340302 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" containerID="cri-o://2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f" gracePeriod=30 Nov 03 20:34:17.602172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.600755 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 20:34:17.603527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:17.602570 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 20:34:19.524437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.524400 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh"] Nov 03 20:34:19.581733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.581702 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh"] Nov 03 20:34:19.581961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.581800 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.586557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.586534 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-14ebbb-openshift-default-dockercfg-cnqz2\"" Nov 03 20:34:19.698083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.698037 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.698472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.698427 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.702759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.702730 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.703011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.702979 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlfz\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.703235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.703206 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.703663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.703642 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.703864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.703847 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.704022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.704004 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.704146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.704132 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.805943 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806014 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlfz\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806206 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806231 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806276 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806368 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806439 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.808572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.806935 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.811543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.809849 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.811543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.810169 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.811543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.810955 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.811543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.811052 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.832200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.828159 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.853547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.851204 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.854063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.854033 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:19.855254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:19.855228 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlfz\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz\") pod \"e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:20.003091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:20.002794 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:20.447381 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:34:20.447349 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6976b5_e1d9_442b_9b8f_a06b6edf290b.slice/crio-ce465c75b442a963baba24df8ed367713dccd1a738a3f02ba8527c5baaaafceb WatchSource:0}: Error finding container ce465c75b442a963baba24df8ed367713dccd1a738a3f02ba8527c5baaaafceb: Status 404 returned error can't find the container with id ce465c75b442a963baba24df8ed367713dccd1a738a3f02ba8527c5baaaafceb Nov 03 20:34:20.449966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:20.449845 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh"] Nov 03 20:34:20.453647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:20.453459 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:20.453647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:20.453570 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:20.453647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:20.453618 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.105617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.100853 5815 generic.go:356] "Generic (PLEG): container finished" podID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerID="8a95ae10a92e560c4b9b641179e4f692f55b4cf88a902cc446a36dd28c78aa23" exitCode=0 Nov 03 20:34:21.106321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.106289 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" event={"ID":"0e375507-e16f-4bb4-8187-39e26e4e5348","Type":"ContainerDied","Data":"8a95ae10a92e560c4b9b641179e4f692f55b4cf88a902cc446a36dd28c78aa23"} Nov 03 20:34:21.117598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.107108 5815 scope.go:122] "RemoveContainer" containerID="8a95ae10a92e560c4b9b641179e4f692f55b4cf88a902cc446a36dd28c78aa23" Nov 03 20:34:21.127701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.127656 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.128037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.128005 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.128264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.128238 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.143176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.141681 5815 generic.go:356] "Generic (PLEG): container finished" podID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerID="1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3" exitCode=0 Nov 03 20:34:21.143176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.141824 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" event={"ID":"346a0e2d-1843-4125-bc93-b6bf25993c3b","Type":"ContainerDied","Data":"1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3"} Nov 03 20:34:21.143176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.142440 5815 scope.go:122] "RemoveContainer" containerID="1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3" Nov 03 20:34:21.160694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.146474 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.160694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.146619 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.160694 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.146666 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:21.230062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.228895 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" event={"ID":"ac6976b5-e1d9-442b-9b8f-a06b6edf290b","Type":"ContainerStarted","Data":"00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264"} Nov 03 20:34:21.230062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.228956 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" event={"ID":"ac6976b5-e1d9-442b-9b8f-a06b6edf290b","Type":"ContainerStarted","Data":"ce465c75b442a963baba24df8ed367713dccd1a738a3f02ba8527c5baaaafceb"} Nov 03 20:34:21.265029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:21.264845 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podStartSLOduration=2.264828685 podStartE2EDuration="2.264828685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:21.260395709 +0000 UTC m=+1970.790741380" watchObservedRunningTime="2026-11-03 20:34:21.264828685 +0000 UTC m=+1970.795174358" Nov 03 20:34:22.013531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.012497 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:22.020056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.019953 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:22.248135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.248103 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" event={"ID":"0e375507-e16f-4bb4-8187-39e26e4e5348","Type":"ContainerStarted","Data":"2631a874f01034ff7ac630971e18a0f77db9d1731551ed7bde4fb5c73e3f12f4"} Nov 03 20:34:22.252044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.252016 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" event={"ID":"346a0e2d-1843-4125-bc93-b6bf25993c3b","Type":"ContainerStarted","Data":"0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388"} Nov 03 20:34:22.252254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.252237 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:22.253405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.253385 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:34:22.459091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.458932 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:34:22.477868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.477838 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:22.478159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.478116 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:22.737572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.737170 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:34:22.870277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.870245 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:34:22.965589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.949232 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:22.977710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.977663 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:22.978428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.978405 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:22.991530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.988791 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:22.991703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:22.991665 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.001678 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.001789 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.001839 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.001890 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.001950 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002011 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002094 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002159 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002267 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002304 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002378 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002439 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert\") pod \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\" (UID: \"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002475 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.002539 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket\") pod \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\" (UID: \"6c65f969-4693-49e8-8b9e-e8049c21a5cc\") " Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.003476 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.003500 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.007774 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.016911 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.017281 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.017700 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.018172 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.018567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.018534 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.019599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.018783 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.019668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.019617 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.022628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.022600 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.025660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.025609 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.026131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.026105 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.026232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.026161 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:23.026301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.026278 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:23.026795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.026771 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58" pod "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" (UID: "1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305"). InnerVolumeSpecName "kube-api-access-v5j58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.027448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.027413 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "kube-api-access-wvkks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:23.029626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.029599 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo" pod "6c65f969-4693-49e8-8b9e-e8049c21a5cc" (UID: "6c65f969-4693-49e8-8b9e-e8049c21a5cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.116952 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.116998 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117013 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117030 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvkks\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-kube-api-access-wvkks\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117048 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117061 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117075 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117104 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117119 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5j58\" (UniqueName: \"kubernetes.io/projected/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-kube-api-access-v5j58\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117133 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117147 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117159 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117172 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117185 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117198 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c65f969-4693-49e8-8b9e-e8049c21a5cc-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.120532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.117212 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c65f969-4693-49e8-8b9e-e8049c21a5cc-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.131652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.130943 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:34:23.131652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.131075 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 20:34:23.131652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.131119 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 20:34:23.441427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.441390 5815 generic.go:356] "Generic (PLEG): container finished" podID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerID="2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f" exitCode=0 Nov 03 20:34:23.441950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.441453 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" Nov 03 20:34:23.441950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.441490 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" event={"ID":"6c65f969-4693-49e8-8b9e-e8049c21a5cc","Type":"ContainerDied","Data":"2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f"} Nov 03 20:34:23.441950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.441603 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz" event={"ID":"6c65f969-4693-49e8-8b9e-e8049c21a5cc","Type":"ContainerDied","Data":"a9b191ceb413dcc5a5b3408582661f9ec522fdb0c4778c189f773805f1f23150"} Nov 03 20:34:23.441950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.441888 5815 scope.go:122] "RemoveContainer" containerID="2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f" Nov 03 20:34:23.444943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.444917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" event={"ID":"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305","Type":"ContainerDied","Data":"c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17"} Nov 03 20:34:23.445046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.444842 5815 generic.go:356] "Generic (PLEG): container finished" podID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerID="c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17" exitCode=0 Nov 03 20:34:23.445046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.444980 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" Nov 03 20:34:23.445046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.444995 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9" event={"ID":"1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305","Type":"ContainerDied","Data":"40712fff507a9909565200f927d5961df38199bab8892d9b095be000662c2e8b"} Nov 03 20:34:23.448706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.448684 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:23.448845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.448731 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:23.462480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.462464 5815 scope.go:122] "RemoveContainer" containerID="2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f" Nov 03 20:34:23.462836 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:34:23.462818 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f\": container with ID starting with 2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f not found: ID does not exist" containerID="2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f" Nov 03 20:34:23.462890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.462846 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f"} err="failed to get container status \"2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f\": rpc error: code = NotFound desc = could not find container \"2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f\": container with ID starting with 2bf513b05e5a83801c5ff032ffa0bb4ca60b2d92a3585afc4fa81714bbea846f not found: ID does not exist" Nov 03 20:34:23.462890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.462863 5815 scope.go:122] "RemoveContainer" containerID="c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17" Nov 03 20:34:23.553364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.553309 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz"] Nov 03 20:34:23.560138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.560108 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-n46jz"] Nov 03 20:34:23.563247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.563222 5815 scope.go:122] "RemoveContainer" containerID="c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17" Nov 03 20:34:23.563812 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:34:23.563746 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17\": container with ID starting with c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17 not found: ID does not exist" containerID="c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17" Nov 03 20:34:23.563913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.563826 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17"} err="failed to get container status \"c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17\": rpc error: code = NotFound desc = could not find container \"c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17\": container with ID starting with c5b5a0c9fccce8e92ef7afae8cff2c408f05d61e250459a36fee070d12fc0f17 not found: ID does not exist" Nov 03 20:34:23.591768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.591726 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9"] Nov 03 20:34:23.630359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:23.630269 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-w7zk9"] Nov 03 20:34:24.134534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:24.130355 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 20:34:24.134534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:24.130415 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 20:34:24.449332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:24.449247 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:24.449332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:24.449317 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:24.645899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:24.645863 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" path="/var/lib/kubelet/pods/1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305/volumes" Nov 03 20:34:24.647420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:24.647390 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" path="/var/lib/kubelet/pods/6c65f969-4693-49e8-8b9e-e8049c21a5cc/volumes" Nov 03 20:34:25.141529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:25.133634 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 20:34:25.141529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:25.133695 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 20:34:25.517113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:25.517081 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:25.535533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:25.533597 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:26.135529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:26.129914 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 20:34:26.135529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:26.129988 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 20:34:26.448995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:26.448913 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:26.448995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:26.448961 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:27.130944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:27.130904 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 20:34:27.131573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:27.131541 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 20:34:27.448933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:27.448859 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:27.449151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:27.449128 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:28.128412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:28.128376 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 20:34:28.128767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:28.128739 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 20:34:28.419589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:28.418933 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:34:28.450899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:28.450862 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 20:34:28.451184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:28.451154 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 20:34:29.321110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:29.321037 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:34:29.322121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:29.322095 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:34:29.323014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:29.322991 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:34:29.562044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:29.562016 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:34:29.564766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:29.564742 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:34:32.684713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:32.684685 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:34:39.317943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.317863 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb"] Nov 03 20:34:39.318673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.318653 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" Nov 03 20:34:39.318673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.318671 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" Nov 03 20:34:39.318823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.318694 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" Nov 03 20:34:39.318823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.318699 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" Nov 03 20:34:39.318823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.318811 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eec6b7c-d10e-4fb6-b6d4-3e7f5cf09305" containerName="istio-proxy" Nov 03 20:34:39.318937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.318831 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c65f969-4693-49e8-8b9e-e8049c21a5cc" containerName="istio-proxy" Nov 03 20:34:39.321948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.321930 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.325951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.325930 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-d84007a46645\"" Nov 03 20:34:39.327249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.327225 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"openshift-service-ca.crt\"" Nov 03 20:34:39.327249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.327251 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"payload-processing-external-model-plugins-e2e-extmod-b-03c633\"" Nov 03 20:34:39.327439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.327258 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"kube-root-ca.crt\"" Nov 03 20:34:39.327439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.327327 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"routing-overlay\"" Nov 03 20:34:39.327439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.327407 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"payload-processing-external-model-e2e--bb65a178-dockercfg-5zp5h\"" Nov 03 20:34:39.329443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.329416 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb"] Nov 03 20:34:39.350322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.350284 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hft5s\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.350459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.350348 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.350579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.350555 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.350783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.350762 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.457531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.457278 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hft5s\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.457531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.457351 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.457531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.457469 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.511532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.499755 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.651582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.633183 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.701538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.684117 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.701538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.684830 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.780537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.779082 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hft5s\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:39.940524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:39.940431 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:40.564858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:40.563092 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb"] Nov 03 20:34:40.926323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:40.926231 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" event={"ID":"6b3ab081-cb43-48f1-b503-6e2b624e8537","Type":"ContainerStarted","Data":"39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668"} Nov 03 20:34:40.926323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:40.926286 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" event={"ID":"6b3ab081-cb43-48f1-b503-6e2b624e8537","Type":"ContainerStarted","Data":"76097466dfc01dfd5eaa55dbf8df321047ceeb12a3b97e9edaa37b56f854540b"} Nov 03 20:34:40.926529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:40.926323 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:40.958059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:40.957995 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" podStartSLOduration=1.957982061 podStartE2EDuration="1.957982061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:40.952833645 +0000 UTC m=+1990.483179317" watchObservedRunningTime="2026-11-03 20:34:40.957982061 +0000 UTC m=+1990.488327733" Nov 03 20:34:43.853893 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:34:43.853858 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:34:46.091166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.090899 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92"] Nov 03 20:34:46.120761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.120731 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92"] Nov 03 20:34:46.120973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.120852 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.124972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.124944 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c7aecb-openshift-default-dockercfg-97wvf\"" Nov 03 20:34:46.236286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.236246 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.236621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.236594 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.237006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.236986 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.237165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.237149 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.237300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.237282 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.237407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.237393 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.242407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.242376 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.242771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.242687 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtjqk\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.243040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.243020 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.354057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.353970 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.355672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.355646 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtjqk\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.356156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.356134 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.356310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.356293 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.356489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.356472 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.356886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.356867 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.372619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.372591 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.372852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.372830 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.372997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.372980 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.402540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.373975 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.409467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.389531 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.409698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.365958 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.409840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.366926 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.409938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.389780 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.417530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.416077 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.452198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.452113 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.460537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.456219 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtjqk\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.488122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.488089 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:46.739860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:46.739821 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:47.135305 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:34:47.135270 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540934a6_2005_4927_b558_441ba575a73d.slice/crio-422c2ac036aac72042ccf9363eb1dc2fa96edb6116f72dbe84361dac79483061 WatchSource:0}: Error finding container 422c2ac036aac72042ccf9363eb1dc2fa96edb6116f72dbe84361dac79483061: Status 404 returned error can't find the container with id 422c2ac036aac72042ccf9363eb1dc2fa96edb6116f72dbe84361dac79483061 Nov 03 20:34:47.136907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:47.136843 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92"] Nov 03 20:34:47.138545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:47.138478 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:47.138637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:47.138567 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:47.138637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:47.138600 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:34:48.071375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:48.071338 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" event={"ID":"540934a6-2005-4927-b558-441ba575a73d","Type":"ContainerStarted","Data":"c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb"} Nov 03 20:34:48.071627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:48.071403 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" event={"ID":"540934a6-2005-4927-b558-441ba575a73d","Type":"ContainerStarted","Data":"422c2ac036aac72042ccf9363eb1dc2fa96edb6116f72dbe84361dac79483061"} Nov 03 20:34:48.098535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:48.098470 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podStartSLOduration=2.098455996 podStartE2EDuration="2.098455996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:48.096028856 +0000 UTC m=+1997.626374530" watchObservedRunningTime="2026-11-03 20:34:48.098455996 +0000 UTC m=+1997.628801668" Nov 03 20:34:48.741561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:48.741486 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:48.748112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:48.748088 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:49.086083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:49.085994 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:49.087071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:49.087047 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:34:51.586651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.586621 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq"] Nov 03 20:34:51.590237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.590220 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.596171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.596137 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c7aecb\"" Nov 03 20:34:51.596300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.596272 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c7aecb-dockercfg-jq9j5\"" Nov 03 20:34:51.605214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.605187 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq"] Nov 03 20:34:51.654379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.654335 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.654597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.654561 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qzs\" (UniqueName: \"kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs\") pod \"payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.779531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.763887 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qzs\" (UniqueName: \"kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs\") pod \"payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.779531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.764171 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.779531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.765237 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.847674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.847598 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qzs\" (UniqueName: \"kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs\") pod \"payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:51.937533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:51.933059 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:52.010053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:52.010026 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:34:52.469756 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:34:52.469725 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cd0dc4_6ccd_4bf7_b158_5dcb1fb7f180.slice/crio-984e2f0dbcaf0ff1865b22bd1e46a63fd87aa8e705ccf3d7a279ab866857e812 WatchSource:0}: Error finding container 984e2f0dbcaf0ff1865b22bd1e46a63fd87aa8e705ccf3d7a279ab866857e812: Status 404 returned error can't find the container with id 984e2f0dbcaf0ff1865b22bd1e46a63fd87aa8e705ccf3d7a279ab866857e812 Nov 03 20:34:52.470226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:52.470206 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq"] Nov 03 20:34:53.256900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:53.256862 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" event={"ID":"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180","Type":"ContainerStarted","Data":"8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040"} Nov 03 20:34:53.257563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:53.257536 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" event={"ID":"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180","Type":"ContainerStarted","Data":"984e2f0dbcaf0ff1865b22bd1e46a63fd87aa8e705ccf3d7a279ab866857e812"} Nov 03 20:34:53.262456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:53.262436 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:34:53.285002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:34:53.284955 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" podStartSLOduration=2.284937881 podStartE2EDuration="2.284937881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:53.283089586 +0000 UTC m=+2002.813435259" watchObservedRunningTime="2026-11-03 20:34:53.284937881 +0000 UTC m=+2002.815283607" Nov 03 20:35:05.277472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:05.277434 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 20:35:05.283315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:05.283286 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:35:06.545374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:06.545310 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:09.059532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.041607 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:09.598548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.594876 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn"] Nov 03 20:35:09.615982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.615957 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:09.650709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.649113 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"agc-cred-d7a3528edd35\"" Nov 03 20:35:09.667356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.667326 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn"] Nov 03 20:35:09.766533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.736288 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rvpt\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:09.856696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.856608 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:09.866706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.866668 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:09.961464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:09.961350 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.092917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.092872 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rvpt\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.093435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.092952 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.093435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.093008 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.093435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.093066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.094059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.094028 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.094464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.094440 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.095988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.095962 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.105334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.105305 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rvpt\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt\") pod \"payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.251391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.251357 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:10.570726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:10.570683 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn"] Nov 03 20:35:10.572306 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:35:10.572282 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b760e2d_c334_46c9_b0d3_4453dce5e70d.slice/crio-27d54ddad11523135c4fb78589aa7828edcdbdf0cabc0e854c100a3ca1cc492d WatchSource:0}: Error finding container 27d54ddad11523135c4fb78589aa7828edcdbdf0cabc0e854c100a3ca1cc492d: Status 404 returned error can't find the container with id 27d54ddad11523135c4fb78589aa7828edcdbdf0cabc0e854c100a3ca1cc492d Nov 03 20:35:11.063113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.063075 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" event={"ID":"1b760e2d-c334-46c9-b0d3-4453dce5e70d","Type":"ContainerStarted","Data":"e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24"} Nov 03 20:35:11.063573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.063550 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" event={"ID":"1b760e2d-c334-46c9-b0d3-4453dce5e70d","Type":"ContainerStarted","Data":"27d54ddad11523135c4fb78589aa7828edcdbdf0cabc0e854c100a3ca1cc492d"} Nov 03 20:35:11.065581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.065560 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:11.248531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.234366 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" podStartSLOduration=2.234346899 podStartE2EDuration="2.234346899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:11.164078894 +0000 UTC m=+2020.694424566" watchObservedRunningTime="2026-11-03 20:35:11.234346899 +0000 UTC m=+2020.764692572" Nov 03 20:35:11.587533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.566005 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6"] Nov 03 20:35:11.622910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.622874 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:11.648651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.632154 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6350b9-dockercfg-hpg2k\"" Nov 03 20:35:11.648651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.632772 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6350b9\"" Nov 03 20:35:11.680835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.680801 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47g9\" (UniqueName: \"kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9\") pod \"payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:11.685364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.685332 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:11.761859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.760822 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6"] Nov 03 20:35:11.802567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.798845 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47g9\" (UniqueName: \"kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9\") pod \"payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:11.802567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.799004 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:11.802567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.800030 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:11.871581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:11.852582 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47g9\" (UniqueName: \"kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9\") pod \"payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:12.015272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:12.015000 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:12.044800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:12.043796 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:35:12.756193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:12.755742 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6"] Nov 03 20:35:13.208537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.207563 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq"] Nov 03 20:35:13.257931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.257628 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.265463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.265376 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-b7e8ea2ec1e9\"" Nov 03 20:35:13.265463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.265428 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-0d43e4ffb5b7\"" Nov 03 20:35:13.266197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.266063 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-e2e--f590b3a5-dockercfg-mjjk7\"" Nov 03 20:35:13.281260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.281228 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq"] Nov 03 20:35:13.288909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.288115 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"kube-root-ca.crt\"" Nov 03 20:35:13.288909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.288255 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-05473b\"" Nov 03 20:35:13.288909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.288262 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"openshift-service-ca.crt\"" Nov 03 20:35:13.288909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.288320 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"routing-overlay\"" Nov 03 20:35:13.293178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.293119 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" event={"ID":"c226212b-ca3d-4b73-8ed5-719fe27c463d","Type":"ContainerStarted","Data":"61cad69aeed77893e17147b2118530d6c52215d1c3384b5173a1309a84dafb88"} Nov 03 20:35:13.351495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.351448 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.351814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.351789 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.351968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.351887 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.352135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.352092 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.453606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.453573 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.453797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.453689 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.453941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.453924 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.454031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.454012 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.454420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.454394 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.454680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.454656 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.456757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.456736 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.466950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.466927 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.610205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.610127 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:13.859676 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:35:13.859632 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:35:13.998922 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:35:13.998890 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb54e9a_31e3_4eb5_81e1_d1009e88f82f.slice/crio-515ab90113f492bb0d42dc56e47cf8e480a7fc0e598847d3ba5a2f3345075bb7 WatchSource:0}: Error finding container 515ab90113f492bb0d42dc56e47cf8e480a7fc0e598847d3ba5a2f3345075bb7: Status 404 returned error can't find the container with id 515ab90113f492bb0d42dc56e47cf8e480a7fc0e598847d3ba5a2f3345075bb7 Nov 03 20:35:13.999365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:13.999339 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq"] Nov 03 20:35:14.304726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.304635 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" event={"ID":"c226212b-ca3d-4b73-8ed5-719fe27c463d","Type":"ContainerStarted","Data":"3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7"} Nov 03 20:35:14.304726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.304723 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:14.306919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.306888 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" event={"ID":"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f","Type":"ContainerStarted","Data":"0cba287ea20edf03ce4b09e6e97b3a24ba88b367867bf376c069fb8cbba7c467"} Nov 03 20:35:14.307061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.306937 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" event={"ID":"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f","Type":"ContainerStarted","Data":"515ab90113f492bb0d42dc56e47cf8e480a7fc0e598847d3ba5a2f3345075bb7"} Nov 03 20:35:14.307061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.306982 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:14.349876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.340027 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" podStartSLOduration=3.340009901 podStartE2EDuration="3.340009901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:14.339113721 +0000 UTC m=+2023.869459394" watchObservedRunningTime="2026-11-03 20:35:14.340009901 +0000 UTC m=+2023.870355574" Nov 03 20:35:14.401544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:14.397798 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" podStartSLOduration=1.397772456 podStartE2EDuration="1.397772456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:14.395965076 +0000 UTC m=+2023.926310739" watchObservedRunningTime="2026-11-03 20:35:14.397772456 +0000 UTC m=+2023.928118165" Nov 03 20:35:23.296020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:23.295985 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:35:23.348332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:23.348299 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt"] Nov 03 20:35:25.324129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:25.324102 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:35:25.324627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:25.324169 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 20:35:28.349611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.349563 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" podUID="f19c76ca-2f6f-44c5-96e1-b13a50acc512" containerName="payload-processing" containerID="cri-o://585a47af37123348438578daad13203f337059b38e696d8c3daf50b21920179b" gracePeriod=25 Nov 03 20:35:28.696894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.696845 5815 generic.go:356] "Generic (PLEG): container finished" podID="f19c76ca-2f6f-44c5-96e1-b13a50acc512" containerID="585a47af37123348438578daad13203f337059b38e696d8c3daf50b21920179b" exitCode=0 Nov 03 20:35:28.697046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.696913 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" event={"ID":"f19c76ca-2f6f-44c5-96e1-b13a50acc512","Type":"ContainerDied","Data":"585a47af37123348438578daad13203f337059b38e696d8c3daf50b21920179b"} Nov 03 20:35:28.697046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.697014 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" event={"ID":"f19c76ca-2f6f-44c5-96e1-b13a50acc512","Type":"ContainerDied","Data":"fed6dbcfb54486769ff60ee86fe4aebe9c4788ea241d033aee803d38bca123e4"} Nov 03 20:35:28.697046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.697037 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed6dbcfb54486769ff60ee86fe4aebe9c4788ea241d033aee803d38bca123e4" Nov 03 20:35:28.700541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.700500 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:35:28.822965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.822933 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume\") pod \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " Nov 03 20:35:28.823171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.822977 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2\") pod \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " Nov 03 20:35:28.823365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.823340 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume\") pod \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " Nov 03 20:35:28.823441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.823382 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume" pod "f19c76ca-2f6f-44c5-96e1-b13a50acc512" (UID: "f19c76ca-2f6f-44c5-96e1-b13a50acc512"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:28.823441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.823433 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials\") pod \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\" (UID: \"f19c76ca-2f6f-44c5-96e1-b13a50acc512\") " Nov 03 20:35:28.823721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.823698 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume" pod "f19c76ca-2f6f-44c5-96e1-b13a50acc512" (UID: "f19c76ca-2f6f-44c5-96e1-b13a50acc512"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:28.824054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.824037 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:28.824112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.824057 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f19c76ca-2f6f-44c5-96e1-b13a50acc512-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:28.842569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.827817 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2" pod "f19c76ca-2f6f-44c5-96e1-b13a50acc512" (UID: "f19c76ca-2f6f-44c5-96e1-b13a50acc512"). InnerVolumeSpecName "kube-api-access-bwlj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:28.842569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.836297 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials" pod "f19c76ca-2f6f-44c5-96e1-b13a50acc512" (UID: "f19c76ca-2f6f-44c5-96e1-b13a50acc512"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:28.925469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.925388 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:28.925469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:28.925420 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwlj2\" (UniqueName: \"kubernetes.io/projected/f19c76ca-2f6f-44c5-96e1-b13a50acc512-kube-api-access-bwlj2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.711466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:29.711431 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt" Nov 03 20:35:29.738621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:29.738592 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt"] Nov 03 20:35:29.754792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:29.754755 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589kvjxt"] Nov 03 20:35:30.725531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:30.721685 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19c76ca-2f6f-44c5-96e1-b13a50acc512" path="/var/lib/kubelet/pods/f19c76ca-2f6f-44c5-96e1-b13a50acc512/volumes" Nov 03 20:35:34.645534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:34.644614 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:34.757531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:34.754175 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:34.987582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:34.983596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:35.065234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.065189 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:35:35.116187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.116145 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:35.147707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.147640 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:35.230924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.230872 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:35:35.374866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.374764 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6"] Nov 03 20:35:35.397189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.377744 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f19c76ca-2f6f-44c5-96e1-b13a50acc512" containerName="payload-processing" Nov 03 20:35:35.397960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.397933 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19c76ca-2f6f-44c5-96e1-b13a50acc512" containerName="payload-processing" Nov 03 20:35:35.398534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.398490 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19c76ca-2f6f-44c5-96e1-b13a50acc512" containerName="payload-processing" Nov 03 20:35:35.413139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.405521 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6"] Nov 03 20:35:35.413139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.405643 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.453108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.435446 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jlpm\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.453108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.435537 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.453108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.435685 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.453108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.435806 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.468907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.458483 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c7aecb\"" Nov 03 20:35:35.468907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.458671 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"payload-processing-external-model-e2e--37db5ca3-dockercfg-v7djd\"" Nov 03 20:35:35.468907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.458730 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"agc-cred-646c4c5fedfb\"" Nov 03 20:35:35.468907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.459391 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"routing-overlay\"" Nov 03 20:35:35.468907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.459493 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"kube-root-ca.crt\"" Nov 03 20:35:35.468907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.459575 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"openshift-service-ca.crt\"" Nov 03 20:35:35.537773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.537739 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.550493 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.553432 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.553601 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.553765 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.553998 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jlpm\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.555231 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.594669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.569801 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jlpm\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:35.660767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:35.660694 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:36.299989 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:35:36.299957 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00611521_41ee_4155_a6c0_adfaf02190b8.slice/crio-1af219f482688c56b2eb2c63d451e43295f4fe357ceb4ac4a28ce1f8ea9eb4e9 WatchSource:0}: Error finding container 1af219f482688c56b2eb2c63d451e43295f4fe357ceb4ac4a28ce1f8ea9eb4e9: Status 404 returned error can't find the container with id 1af219f482688c56b2eb2c63d451e43295f4fe357ceb4ac4a28ce1f8ea9eb4e9 Nov 03 20:35:36.303047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:36.303019 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6"] Nov 03 20:35:36.518476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:36.518444 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" event={"ID":"00611521-41ee-4155-a6c0-adfaf02190b8","Type":"ContainerStarted","Data":"e2e0ceb5e8771151a8347484392f7632717aea8662e46631a8b27285f633fc91"} Nov 03 20:35:36.518740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:36.518547 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" event={"ID":"00611521-41ee-4155-a6c0-adfaf02190b8","Type":"ContainerStarted","Data":"1af219f482688c56b2eb2c63d451e43295f4fe357ceb4ac4a28ce1f8ea9eb4e9"} Nov 03 20:35:36.518740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:36.518682 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:36.544379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:36.544273 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" podStartSLOduration=2.544257265 podStartE2EDuration="2.544257265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:36.53695361 +0000 UTC m=+2046.067299286" watchObservedRunningTime="2026-11-03 20:35:36.544257265 +0000 UTC m=+2046.074602938" Nov 03 20:35:43.894416 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:35:43.894372 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:35:46.770530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:46.765044 5815 generic.go:356] "Generic (PLEG): container finished" podID="540934a6-2005-4927-b558-441ba575a73d" containerID="c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb" exitCode=0 Nov 03 20:35:46.770530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:46.765132 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" event={"ID":"540934a6-2005-4927-b558-441ba575a73d","Type":"ContainerDied","Data":"c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb"} Nov 03 20:35:46.770530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:46.765764 5815 scope.go:122] "RemoveContainer" containerID="c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb" Nov 03 20:35:46.770530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:46.768947 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:35:46.770530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:46.769044 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:35:46.770530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:46.769092 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:35:47.531422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:47.531387 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:35:47.827448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:47.827346 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" event={"ID":"540934a6-2005-4927-b558-441ba575a73d","Type":"ContainerStarted","Data":"6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e"} Nov 03 20:35:48.742253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:48.742219 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:35:48.743395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:48.743365 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 20:35:48.743556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:48.743420 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 20:35:49.089717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:49.089602 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:35:49.742116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:49.742084 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 20:35:49.742322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:49.742136 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 20:35:50.744684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:50.744639 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 20:35:50.745371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:50.745330 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 20:35:51.741334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:51.741295 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 20:35:51.741555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:51.741355 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 20:35:52.741678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:52.741638 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Nov 03 20:35:52.742218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:52.741703 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Nov 03 20:35:53.798321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:53.798288 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:35:53.803430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:35:53.803402 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:36:08.097793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:08.097756 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:36:12.031612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:12.031522 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp"] Nov 03 20:36:12.046796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:12.046766 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75ca193e-h9gkp"] Nov 03 20:36:12.591174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:12.591135 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="982157e2-be88-458d-b66b-3caab90b7141" path="/var/lib/kubelet/pods/982157e2-be88-458d-b66b-3caab90b7141/volumes" Nov 03 20:36:13.855706 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:13.855647 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:36:15.107467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.107431 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks"] Nov 03 20:36:15.156971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.156944 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.177598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.177429 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"agc-cred-324faedf197f\"" Nov 03 20:36:15.180169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.156820 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks"] Nov 03 20:36:15.432430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.354014 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.476251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.472366 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.476251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.472419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4xrb\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.476251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.472459 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.648742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.648695 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh"] Nov 03 20:36:15.669104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.669066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.669315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.669133 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.669315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.669154 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4xrb\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.669315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.669181 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.672081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.672054 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.672449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.672426 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.673278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.673261 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.675217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.675192 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.677001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.676982 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-343c202a1e2b\"" Nov 03 20:36:15.678470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.678446 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh"] Nov 03 20:36:15.682979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.682899 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4xrb\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb\") pod \"payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.826959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.826926 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:15.862319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.851984 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sgj\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.863083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.863049 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.863251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.863193 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.864908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.864885 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.997530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.973716 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.997530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.973825 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sgj\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.997530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.973958 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.997530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.974066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.997530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.981195 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:15.997530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:15.981903 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:16.016675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.016590 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:16.180554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.178875 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sgj\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:16.181568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.181288 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:16.719884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.714457 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks"] Nov 03 20:36:16.909409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.909376 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" event={"ID":"ebe2868b-d806-4933-bb12-8b64ce4aadb9","Type":"ContainerStarted","Data":"06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059"} Nov 03 20:36:16.909630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.909438 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" event={"ID":"ebe2868b-d806-4933-bb12-8b64ce4aadb9","Type":"ContainerStarted","Data":"e0b525b6803862c01781eb433686206752cfc5f9b93814c1755241978368e886"} Nov 03 20:36:16.909630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.909477 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:16.929391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.929342 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" podStartSLOduration=1.929330845 podStartE2EDuration="1.929330845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:16.92873162 +0000 UTC m=+2086.459077290" watchObservedRunningTime="2026-11-03 20:36:16.929330845 +0000 UTC m=+2086.459676517" Nov 03 20:36:16.979989 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:36:16.979960 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335b171b_9980_4593_8806_1b95b1ab7fd3.slice/crio-af4e7f44c8164236cd88e564be18f870d14db26d177b5ef72f9eaa2982c79b55 WatchSource:0}: Error finding container af4e7f44c8164236cd88e564be18f870d14db26d177b5ef72f9eaa2982c79b55: Status 404 returned error can't find the container with id af4e7f44c8164236cd88e564be18f870d14db26d177b5ef72f9eaa2982c79b55 Nov 03 20:36:16.981902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:16.981875 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh"] Nov 03 20:36:17.920707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:17.920667 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" event={"ID":"335b171b-9980-4593-8806-1b95b1ab7fd3","Type":"ContainerStarted","Data":"a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8"} Nov 03 20:36:17.921216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:17.920727 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:17.921216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:17.920748 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" event={"ID":"335b171b-9980-4593-8806-1b95b1ab7fd3","Type":"ContainerStarted","Data":"af4e7f44c8164236cd88e564be18f870d14db26d177b5ef72f9eaa2982c79b55"} Nov 03 20:36:17.942084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:17.942034 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" podStartSLOduration=2.942020489 podStartE2EDuration="2.942020489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:17.940601702 +0000 UTC m=+2087.470947375" watchObservedRunningTime="2026-11-03 20:36:17.942020489 +0000 UTC m=+2087.472366160" Nov 03 20:36:18.026046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:18.026006 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs"] Nov 03 20:36:18.030918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:18.030892 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e345ebc-bwdhs"] Nov 03 20:36:18.589004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:18.588961 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367fcb0b-09d4-4aec-801a-33444de28026" path="/var/lib/kubelet/pods/367fcb0b-09d4-4aec-801a-33444de28026/volumes" Nov 03 20:36:22.085535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:22.081572 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq"] Nov 03 20:36:22.114549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:22.111635 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a668a642-hhfcq"] Nov 03 20:36:22.593386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:22.593347 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee56785e-4c8a-40e7-8347-07c9413f245f" path="/var/lib/kubelet/pods/ee56785e-4c8a-40e7-8347-07c9413f245f/volumes" Nov 03 20:36:23.034256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:23.034215 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks"] Nov 03 20:36:23.063089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:23.063062 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:25.537965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.537926 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr"] Nov 03 20:36:25.543046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.543025 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.547009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.546985 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:25.548079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.548058 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:36:25.552039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.552016 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr"] Nov 03 20:36:25.574366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.574338 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.575491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.575464 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssvcr\" (UniqueName: \"kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.575868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.575847 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.713529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.704579 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.713529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.705028 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssvcr\" (UniqueName: \"kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.713529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.705403 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.713529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.709471 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.743532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.736804 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssvcr\" (UniqueName: \"kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.743532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.737020 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95772dc-4gsjr\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:25.862579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:25.862487 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:26.163917 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:36:26.163888 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd1f003_df61_4b5a_b5db_5dd8c493f860.slice/crio-28d809e6abd25c2994ad514c6bff161dd4bc9ad888f94ed2b7b30409affae62c WatchSource:0}: Error finding container 28d809e6abd25c2994ad514c6bff161dd4bc9ad888f94ed2b7b30409affae62c: Status 404 returned error can't find the container with id 28d809e6abd25c2994ad514c6bff161dd4bc9ad888f94ed2b7b30409affae62c Nov 03 20:36:26.165881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:26.165849 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr"] Nov 03 20:36:27.132099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:27.132065 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" event={"ID":"bfd1f003-df61-4b5a-b5db-5dd8c493f860","Type":"ContainerStarted","Data":"3de1c338cf428979b4e0c2077b422069ae5345b9bc9805248c240ff1d5400d95"} Nov 03 20:36:27.132646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:27.132127 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" event={"ID":"bfd1f003-df61-4b5a-b5db-5dd8c493f860","Type":"ContainerStarted","Data":"28d809e6abd25c2994ad514c6bff161dd4bc9ad888f94ed2b7b30409affae62c"} Nov 03 20:36:27.155882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:27.155834 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" podStartSLOduration=2.155819996 podStartE2EDuration="2.155819996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:27.150988994 +0000 UTC m=+2096.681334667" watchObservedRunningTime="2026-11-03 20:36:27.155819996 +0000 UTC m=+2096.686165668" Nov 03 20:36:28.034760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.034700 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" podUID="ebe2868b-d806-4933-bb12-8b64ce4aadb9" containerName="payload-processing" containerID="cri-o://06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059" gracePeriod=25 Nov 03 20:36:28.142977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.142940 5815 generic.go:356] "Generic (PLEG): container finished" podID="bfd1f003-df61-4b5a-b5db-5dd8c493f860" containerID="3de1c338cf428979b4e0c2077b422069ae5345b9bc9805248c240ff1d5400d95" exitCode=0 Nov 03 20:36:28.143352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.143019 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" event={"ID":"bfd1f003-df61-4b5a-b5db-5dd8c493f860","Type":"ContainerDied","Data":"3de1c338cf428979b4e0c2077b422069ae5345b9bc9805248c240ff1d5400d95"} Nov 03 20:36:28.358352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.358325 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:28.464226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.464197 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials\") pod \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " Nov 03 20:36:28.464462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.464277 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume\") pod \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " Nov 03 20:36:28.464462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.464305 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb\") pod \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " Nov 03 20:36:28.464462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.464404 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume\") pod \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\" (UID: \"ebe2868b-d806-4933-bb12-8b64ce4aadb9\") " Nov 03 20:36:28.464791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.464766 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume" pod "ebe2868b-d806-4933-bb12-8b64ce4aadb9" (UID: "ebe2868b-d806-4933-bb12-8b64ce4aadb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:28.464888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.464864 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume" pod "ebe2868b-d806-4933-bb12-8b64ce4aadb9" (UID: "ebe2868b-d806-4933-bb12-8b64ce4aadb9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:28.465687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.465662 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:28.465780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.465691 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebe2868b-d806-4933-bb12-8b64ce4aadb9-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:28.467122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.467086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials" pod "ebe2868b-d806-4933-bb12-8b64ce4aadb9" (UID: "ebe2868b-d806-4933-bb12-8b64ce4aadb9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:28.472696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.472667 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb" pod "ebe2868b-d806-4933-bb12-8b64ce4aadb9" (UID: "ebe2868b-d806-4933-bb12-8b64ce4aadb9"). InnerVolumeSpecName "kube-api-access-x4xrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:28.580033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.577569 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:28.580033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.577608 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4xrb\" (UniqueName: \"kubernetes.io/projected/ebe2868b-d806-4933-bb12-8b64ce4aadb9-kube-api-access-x4xrb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:28.933269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.933187 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:36:28.999346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:28.999301 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq"] Nov 03 20:36:29.220165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.220130 5815 generic.go:356] "Generic (PLEG): container finished" podID="ebe2868b-d806-4933-bb12-8b64ce4aadb9" containerID="06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059" exitCode=0 Nov 03 20:36:29.221227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.220781 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" event={"ID":"ebe2868b-d806-4933-bb12-8b64ce4aadb9","Type":"ContainerDied","Data":"06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059"} Nov 03 20:36:29.221474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.221455 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" event={"ID":"ebe2868b-d806-4933-bb12-8b64ce4aadb9","Type":"ContainerDied","Data":"e0b525b6803862c01781eb433686206752cfc5f9b93814c1755241978368e886"} Nov 03 20:36:29.222700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.220815 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks" Nov 03 20:36:29.222889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.221547 5815 scope.go:122] "RemoveContainer" containerID="06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059" Nov 03 20:36:29.241284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.241267 5815 scope.go:122] "RemoveContainer" containerID="06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059" Nov 03 20:36:29.241709 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:29.241686 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059\": container with ID starting with 06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059 not found: ID does not exist" containerID="06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059" Nov 03 20:36:29.241825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.241715 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059"} err="failed to get container status \"06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059\": rpc error: code = NotFound desc = could not find container \"06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059\": container with ID starting with 06161a1a67a0b7e609d2db0e1fb729347fd370e5a22e2b825c878b6d727da059 not found: ID does not exist" Nov 03 20:36:29.245991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.245965 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks"] Nov 03 20:36:29.251044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.251021 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-f86b9x9ks"] Nov 03 20:36:29.406962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.406937 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:29.520007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.519923 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca\") pod \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " Nov 03 20:36:29.520175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.520156 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca\") pod \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " Nov 03 20:36:29.520273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.520205 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr\" (UniqueName: \"kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr\") pod \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\" (UID: \"bfd1f003-df61-4b5a-b5db-5dd8c493f860\") " Nov 03 20:36:29.520367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.520338 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca" pod "bfd1f003-df61-4b5a-b5db-5dd8c493f860" (UID: "bfd1f003-df61-4b5a-b5db-5dd8c493f860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:29.520743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.520721 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:29.523051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.523015 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca" pod "bfd1f003-df61-4b5a-b5db-5dd8c493f860" (UID: "bfd1f003-df61-4b5a-b5db-5dd8c493f860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:29.524332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.524311 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr" pod "bfd1f003-df61-4b5a-b5db-5dd8c493f860" (UID: "bfd1f003-df61-4b5a-b5db-5dd8c493f860"). InnerVolumeSpecName "kube-api-access-ssvcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:29.547831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.547802 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb"] Nov 03 20:36:29.622793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.622764 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd1f003-df61-4b5a-b5db-5dd8c493f860-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:29.622793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:29.622792 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssvcr\" (UniqueName: \"kubernetes.io/projected/bfd1f003-df61-4b5a-b5db-5dd8c493f860-kube-api-access-ssvcr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:30.233136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:30.233110 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" Nov 03 20:36:30.233617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:30.233103 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr" event={"ID":"bfd1f003-df61-4b5a-b5db-5dd8c493f860","Type":"ContainerDied","Data":"28d809e6abd25c2994ad514c6bff161dd4bc9ad888f94ed2b7b30409affae62c"} Nov 03 20:36:30.233617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:30.233255 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d809e6abd25c2994ad514c6bff161dd4bc9ad888f94ed2b7b30409affae62c" Nov 03 20:36:30.637940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:30.637858 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe2868b-d806-4933-bb12-8b64ce4aadb9" path="/var/lib/kubelet/pods/ebe2868b-d806-4933-bb12-8b64ce4aadb9/volumes" Nov 03 20:36:34.000641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.000596 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" podUID="ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" containerName="payload-processing" containerID="cri-o://0cba287ea20edf03ce4b09e6e97b3a24ba88b367867bf376c069fb8cbba7c467" gracePeriod=25 Nov 03 20:36:34.286811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.286724 5815 generic.go:356] "Generic (PLEG): container finished" podID="ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" containerID="0cba287ea20edf03ce4b09e6e97b3a24ba88b367867bf376c069fb8cbba7c467" exitCode=0 Nov 03 20:36:34.287049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.287029 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" event={"ID":"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f","Type":"ContainerDied","Data":"0cba287ea20edf03ce4b09e6e97b3a24ba88b367867bf376c069fb8cbba7c467"} Nov 03 20:36:34.324937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.324909 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:36:34.444036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.443993 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume\") pod \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " Nov 03 20:36:34.444459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.444296 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume\") pod \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " Nov 03 20:36:34.444459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.444384 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg\") pod \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " Nov 03 20:36:34.444459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.444463 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials\") pod \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\" (UID: \"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f\") " Nov 03 20:36:34.485563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.446671 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume" pod "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" (UID: "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:34.485563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.446962 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume" pod "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" (UID: "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:34.485563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.450606 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials" pod "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" (UID: "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:34.498589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.493168 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg" pod "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" (UID: "ddb54e9a-31e3-4eb5-81e1-d1009e88f82f"). InnerVolumeSpecName "kube-api-access-4sgdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:34.547903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.547826 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:34.547903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.547864 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:34.547903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.547879 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-kube-api-access-4sgdg\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:34.548124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.547924 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:34.548486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.548455 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" podUID="6b3ab081-cb43-48f1-b503-6e2b624e8537" containerName="payload-processing" containerID="cri-o://39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668" gracePeriod=25 Nov 03 20:36:34.838875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.838850 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:36:34.953259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.953228 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume\") pod \"6b3ab081-cb43-48f1-b503-6e2b624e8537\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " Nov 03 20:36:34.953478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.953462 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume\") pod \"6b3ab081-cb43-48f1-b503-6e2b624e8537\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " Nov 03 20:36:34.953627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.953610 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials\") pod \"6b3ab081-cb43-48f1-b503-6e2b624e8537\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " Nov 03 20:36:34.953698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.953625 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume" pod "6b3ab081-cb43-48f1-b503-6e2b624e8537" (UID: "6b3ab081-cb43-48f1-b503-6e2b624e8537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:34.953757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.953707 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s\") pod \"6b3ab081-cb43-48f1-b503-6e2b624e8537\" (UID: \"6b3ab081-cb43-48f1-b503-6e2b624e8537\") " Nov 03 20:36:34.953820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.953807 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume" pod "6b3ab081-cb43-48f1-b503-6e2b624e8537" (UID: "6b3ab081-cb43-48f1-b503-6e2b624e8537"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:34.954195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.954172 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:34.954284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.954199 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b3ab081-cb43-48f1-b503-6e2b624e8537-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:34.956031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.956008 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials" pod "6b3ab081-cb43-48f1-b503-6e2b624e8537" (UID: "6b3ab081-cb43-48f1-b503-6e2b624e8537"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:34.956031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:34.956021 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s" pod "6b3ab081-cb43-48f1-b503-6e2b624e8537" (UID: "6b3ab081-cb43-48f1-b503-6e2b624e8537"). InnerVolumeSpecName "kube-api-access-hft5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:35.050992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.050963 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:36:35.051384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.051230 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:36:35.055461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.055432 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:35.055461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.055466 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hft5s\" (UniqueName: \"kubernetes.io/projected/6b3ab081-cb43-48f1-b503-6e2b624e8537-kube-api-access-hft5s\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:35.057320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.057300 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:36:35.057465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.057436 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:36:35.345532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.324960 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" event={"ID":"ddb54e9a-31e3-4eb5-81e1-d1009e88f82f","Type":"ContainerDied","Data":"515ab90113f492bb0d42dc56e47cf8e480a7fc0e598847d3ba5a2f3345075bb7"} Nov 03 20:36:35.345532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.325028 5815 scope.go:122] "RemoveContainer" containerID="0cba287ea20edf03ce4b09e6e97b3a24ba88b367867bf376c069fb8cbba7c467" Nov 03 20:36:35.345532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.325625 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq" Nov 03 20:36:35.350903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.350874 5815 generic.go:356] "Generic (PLEG): container finished" podID="6b3ab081-cb43-48f1-b503-6e2b624e8537" containerID="39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668" exitCode=0 Nov 03 20:36:35.351351 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.351197 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" Nov 03 20:36:35.352281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.351216 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" event={"ID":"6b3ab081-cb43-48f1-b503-6e2b624e8537","Type":"ContainerDied","Data":"39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668"} Nov 03 20:36:35.352456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.352439 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb" event={"ID":"6b3ab081-cb43-48f1-b503-6e2b624e8537","Type":"ContainerDied","Data":"76097466dfc01dfd5eaa55dbf8df321047ceeb12a3b97e9edaa37b56f854540b"} Nov 03 20:36:35.373955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.369836 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq"] Nov 03 20:36:35.373955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.373106 5815 scope.go:122] "RemoveContainer" containerID="39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668" Nov 03 20:36:35.377734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.377310 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48bpfbkq"] Nov 03 20:36:35.389568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.388106 5815 scope.go:122] "RemoveContainer" containerID="39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668" Nov 03 20:36:35.389568 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:35.388587 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668\": container with ID starting with 39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668 not found: ID does not exist" containerID="39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668" Nov 03 20:36:35.389568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.388626 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668"} err="failed to get container status \"39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668\": rpc error: code = NotFound desc = could not find container \"39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668\": container with ID starting with 39836b53581a5a621e43154278f695981353c57d835d60b6c9f52ce45b14b668 not found: ID does not exist" Nov 03 20:36:35.393746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.393724 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb"] Nov 03 20:36:35.399212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:35.399190 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc97pllb"] Nov 03 20:36:36.027141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.027110 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx"] Nov 03 20:36:36.032672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.032642 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7223b4-bjrkx"] Nov 03 20:36:36.037636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.037611 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c"] Nov 03 20:36:36.041573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.041554 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0312e9e3-jh68c"] Nov 03 20:36:36.590567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.590533 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cb6436-8fdd-4c9a-a9c5-72f192c93c06" path="/var/lib/kubelet/pods/16cb6436-8fdd-4c9a-a9c5-72f192c93c06/volumes" Nov 03 20:36:36.592206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.592177 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3ab081-cb43-48f1-b503-6e2b624e8537" path="/var/lib/kubelet/pods/6b3ab081-cb43-48f1-b503-6e2b624e8537/volumes" Nov 03 20:36:36.592905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.592884 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb942411-d605-46bb-8d86-06e12a8bd7b8" path="/var/lib/kubelet/pods/cb942411-d605-46bb-8d86-06e12a8bd7b8/volumes" Nov 03 20:36:36.593964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:36.593941 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" path="/var/lib/kubelet/pods/ddb54e9a-31e3-4eb5-81e1-d1009e88f82f/volumes" Nov 03 20:36:37.456560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.456530 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq"] Nov 03 20:36:37.457687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.457666 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6b3ab081-cb43-48f1-b503-6e2b624e8537" containerName="payload-processing" Nov 03 20:36:37.457832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.457822 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3ab081-cb43-48f1-b503-6e2b624e8537" containerName="payload-processing" Nov 03 20:36:37.457892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.457886 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" containerName="payload-processing" Nov 03 20:36:37.457943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.457937 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" containerName="payload-processing" Nov 03 20:36:37.458006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458000 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bfd1f003-df61-4b5a-b5db-5dd8c493f860" containerName="revoke-keys" Nov 03 20:36:37.458055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458048 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd1f003-df61-4b5a-b5db-5dd8c493f860" containerName="revoke-keys" Nov 03 20:36:37.458112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458106 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ebe2868b-d806-4933-bb12-8b64ce4aadb9" containerName="payload-processing" Nov 03 20:36:37.458160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458153 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe2868b-d806-4933-bb12-8b64ce4aadb9" containerName="payload-processing" Nov 03 20:36:37.458354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458345 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb54e9a-31e3-4eb5-81e1-d1009e88f82f" containerName="payload-processing" Nov 03 20:36:37.458417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458411 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd1f003-df61-4b5a-b5db-5dd8c493f860" containerName="revoke-keys" Nov 03 20:36:37.458473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458467 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe2868b-d806-4933-bb12-8b64ce4aadb9" containerName="payload-processing" Nov 03 20:36:37.458560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.458553 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3ab081-cb43-48f1-b503-6e2b624e8537" containerName="payload-processing" Nov 03 20:36:37.463986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.463965 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq"] Nov 03 20:36:37.464120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.464066 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.474963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.468427 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279"] Nov 03 20:36:37.476962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.476650 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279"] Nov 03 20:36:37.476962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.476747 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.483586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.482390 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:36:37.483586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.482734 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:36:37.539164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539140 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.539322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539188 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgw6\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.539322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539214 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.539437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539365 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.539494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539460 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.539574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539553 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.539714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.539700 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.641093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641064 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.641605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641119 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.641605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641171 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgw6\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.641605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641216 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.641605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641298 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.641605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641355 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.641605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641457 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.642000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.641984 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.642320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.642294 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.642648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.642456 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.644216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.644194 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.644373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.644350 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.653556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.653490 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgw6\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6\") pod \"payload-processing-external-model-e2e-extmod-b-421604-5589pblgq\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.653644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.653558 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt\") pod \"maas-api-revoke-sub-4a217c30-xj279\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:37.786685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.786648 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:37.795870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:37.795843 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:38.090680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.070801 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk"] Nov 03 20:36:38.115931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.115911 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5fb11b-28plk"] Nov 03 20:36:38.125714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.121879 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq"] Nov 03 20:36:38.128829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.128717 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279"] Nov 03 20:36:38.473568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.473537 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" event={"ID":"b0425a08-2b24-443d-8d3f-a182414d94ed","Type":"ContainerStarted","Data":"092494ca78e7295f3feaf65613052a57e6c762bb13ce9aec88fc919611b8bd99"} Nov 03 20:36:38.475661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.475631 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" event={"ID":"41d5ae05-5321-4a20-938b-c4dde8cd0ee5","Type":"ContainerStarted","Data":"e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99"} Nov 03 20:36:38.475828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.475687 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" event={"ID":"41d5ae05-5321-4a20-938b-c4dde8cd0ee5","Type":"ContainerStarted","Data":"81a26bcb4548f961e4a209ccd2842e304d81d47227b050875751ebeb342987f9"} Nov 03 20:36:38.475828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.475739 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:38.500323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.500268 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" podStartSLOduration=1.5002517850000001 podStartE2EDuration="1.500251785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:38.494062682 +0000 UTC m=+2108.024408355" watchObservedRunningTime="2026-11-03 20:36:38.500251785 +0000 UTC m=+2108.030597457" Nov 03 20:36:38.591218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:38.591193 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49fcc210-5af4-4b07-b2f7-4821c98866a5" path="/var/lib/kubelet/pods/49fcc210-5af4-4b07-b2f7-4821c98866a5/volumes" Nov 03 20:36:39.592432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:39.592392 5815 generic.go:356] "Generic (PLEG): container finished" podID="b0425a08-2b24-443d-8d3f-a182414d94ed" containerID="71d2ba0face54bcda5f5eb57f0a4c805987f2f128eb1df4b2b761d7368b226bb" exitCode=0 Nov 03 20:36:39.594242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:39.594209 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" event={"ID":"b0425a08-2b24-443d-8d3f-a182414d94ed","Type":"ContainerDied","Data":"71d2ba0face54bcda5f5eb57f0a4c805987f2f128eb1df4b2b761d7368b226bb"} Nov 03 20:36:40.747969 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:40.747939 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-421604/agc-cred-d7a3528edd35: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:40.747969 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:40.747970 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:40.748546 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:40.748068 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials podName:1b760e2d-c334-46c9-b0d3-4453dce5e70d nodeName:}" failed. No retries permitted until 2026-11-03 20:36:41.248046108 +0000 UTC m=+2110.778391781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d") : secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:40.807269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:40.807239 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:40.964129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:40.954037 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca\") pod \"b0425a08-2b24-443d-8d3f-a182414d94ed\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " Nov 03 20:36:40.964129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:40.954302 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt\") pod \"b0425a08-2b24-443d-8d3f-a182414d94ed\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " Nov 03 20:36:40.964129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:40.954401 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca\") pod \"b0425a08-2b24-443d-8d3f-a182414d94ed\" (UID: \"b0425a08-2b24-443d-8d3f-a182414d94ed\") " Nov 03 20:36:40.964129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:40.956551 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca" pod "b0425a08-2b24-443d-8d3f-a182414d94ed" (UID: "b0425a08-2b24-443d-8d3f-a182414d94ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:40.968528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:40.966024 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt" pod "b0425a08-2b24-443d-8d3f-a182414d94ed" (UID: "b0425a08-2b24-443d-8d3f-a182414d94ed"). InnerVolumeSpecName "kube-api-access-rp9nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:41.030002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.029959 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca" pod "b0425a08-2b24-443d-8d3f-a182414d94ed" (UID: "b0425a08-2b24-443d-8d3f-a182414d94ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:41.056389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.056361 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:41.056389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.056387 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/b0425a08-2b24-443d-8d3f-a182414d94ed-kube-api-access-rp9nt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:41.056664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.056398 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0425a08-2b24-443d-8d3f-a182414d94ed-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:41.259343 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:41.259260 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-421604/agc-cred-d7a3528edd35: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:41.259343 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:41.259287 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:41.259343 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:41.259336 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials podName:1b760e2d-c334-46c9-b0d3-4453dce5e70d nodeName:}" failed. No retries permitted until 2026-11-03 20:36:42.259323768 +0000 UTC m=+2111.789669418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d") : secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:41.625740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.625661 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" Nov 03 20:36:41.625740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.625688 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279" event={"ID":"b0425a08-2b24-443d-8d3f-a182414d94ed","Type":"ContainerDied","Data":"092494ca78e7295f3feaf65613052a57e6c762bb13ce9aec88fc919611b8bd99"} Nov 03 20:36:41.625740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:41.625736 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092494ca78e7295f3feaf65613052a57e6c762bb13ce9aec88fc919611b8bd99" Nov 03 20:36:42.277016 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:42.276984 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-421604/agc-cred-d7a3528edd35: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:42.277016 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:42.277016 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:42.277728 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:42.277085 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials podName:1b760e2d-c334-46c9-b0d3-4453dce5e70d nodeName:}" failed. No retries permitted until 2026-11-03 20:36:44.277065424 +0000 UTC m=+2113.807411092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d") : secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:43.854041 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:43.854004 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:36:44.326592 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:44.326557 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-421604/agc-cred-d7a3528edd35: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:44.326787 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:44.326604 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:44.326787 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:44.326674 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials podName:1b760e2d-c334-46c9-b0d3-4453dce5e70d nodeName:}" failed. No retries permitted until 2026-11-03 20:36:48.326658948 +0000 UTC m=+2117.857004597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d") : secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:46.026686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:46.026650 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w"] Nov 03 20:36:46.033775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:46.033656 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620191e8-fvs5w"] Nov 03 20:36:46.593332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:46.593299 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670c716b-b0f2-463d-996f-3c67285e09a9" path="/var/lib/kubelet/pods/670c716b-b0f2-463d-996f-3c67285e09a9/volumes" Nov 03 20:36:47.032347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:47.032272 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9"] Nov 03 20:36:47.052683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:47.052643 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef6eaa03-9jbw9"] Nov 03 20:36:48.036001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.035964 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9"] Nov 03 20:36:48.040710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.040683 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51730991-44wd9"] Nov 03 20:36:48.358267 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:48.358189 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-421604/agc-cred-d7a3528edd35: secret "agc-cred-d7a3528edd35" not found Nov 03 20:36:48.358498 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:48.358485 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-421604/agc-cred-eea7befa4349: secret "agc-cred-eea7befa4349" not found Nov 03 20:36:48.358674 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:48.358628 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn: [secret "agc-cred-d7a3528edd35" not found, secret "agc-cred-eea7befa4349" not found] Nov 03 20:36:48.358869 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:48.358854 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials podName:1b760e2d-c334-46c9-b0d3-4453dce5e70d nodeName:}" failed. No retries permitted until 2026-11-03 20:36:56.358832031 +0000 UTC m=+2125.889177702 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d") : [secret "agc-cred-d7a3528edd35" not found, secret "agc-cred-eea7befa4349" not found] Nov 03 20:36:48.360703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.360662 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq"] Nov 03 20:36:48.365704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.365665 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:48.368882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.368858 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn"] Nov 03 20:36:48.664797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.664699 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b008bd53-0d87-4f40-890f-1ab3384913d8" path="/var/lib/kubelet/pods/b008bd53-0d87-4f40-890f-1ab3384913d8/volumes" Nov 03 20:36:48.665814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:48.665788 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c553e221-b343-41d6-b161-df2219a0f15d" path="/var/lib/kubelet/pods/c553e221-b343-41d6-b161-df2219a0f15d/volumes" Nov 03 20:36:53.362044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.362002 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" podUID="41d5ae05-5321-4a20-938b-c4dde8cd0ee5" containerName="payload-processing" containerID="cri-o://e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99" gracePeriod=25 Nov 03 20:36:53.369584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.369486 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" podUID="1b760e2d-c334-46c9-b0d3-4453dce5e70d" containerName="payload-processing" containerID="cri-o://e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24" gracePeriod=25 Nov 03 20:36:53.780661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.780594 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:53.818637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.818581 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:36:53.819357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.819028 5815 generic.go:356] "Generic (PLEG): container finished" podID="41d5ae05-5321-4a20-938b-c4dde8cd0ee5" containerID="e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99" exitCode=0 Nov 03 20:36:53.819357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.819072 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" Nov 03 20:36:53.819357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.819156 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" event={"ID":"41d5ae05-5321-4a20-938b-c4dde8cd0ee5","Type":"ContainerDied","Data":"e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99"} Nov 03 20:36:53.819357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.819197 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq" event={"ID":"41d5ae05-5321-4a20-938b-c4dde8cd0ee5","Type":"ContainerDied","Data":"81a26bcb4548f961e4a209ccd2842e304d81d47227b050875751ebeb342987f9"} Nov 03 20:36:53.819357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.819223 5815 scope.go:122] "RemoveContainer" containerID="e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99" Nov 03 20:36:53.822097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.821858 5815 generic.go:356] "Generic (PLEG): container finished" podID="1b760e2d-c334-46c9-b0d3-4453dce5e70d" containerID="e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24" exitCode=0 Nov 03 20:36:53.822097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.821898 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" Nov 03 20:36:53.822097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.822000 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" event={"ID":"1b760e2d-c334-46c9-b0d3-4453dce5e70d","Type":"ContainerDied","Data":"e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24"} Nov 03 20:36:53.822097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.822034 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn" event={"ID":"1b760e2d-c334-46c9-b0d3-4453dce5e70d","Type":"ContainerDied","Data":"27d54ddad11523135c4fb78589aa7828edcdbdf0cabc0e854c100a3ca1cc492d"} Nov 03 20:36:53.838462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.838437 5815 scope.go:122] "RemoveContainer" containerID="e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99" Nov 03 20:36:53.844334 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:53.841671 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99\": container with ID starting with e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99 not found: ID does not exist" containerID="e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99" Nov 03 20:36:53.844334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.841712 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99"} err="failed to get container status \"e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99\": rpc error: code = NotFound desc = could not find container \"e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99\": container with ID starting with e8b61a30e18a901b00092c1a3c7026aca081da2837dd04f28e36d1283b7f8b99 not found: ID does not exist" Nov 03 20:36:53.844334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.841741 5815 scope.go:122] "RemoveContainer" containerID="e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24" Nov 03 20:36:53.860707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.860681 5815 scope.go:122] "RemoveContainer" containerID="e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24" Nov 03 20:36:53.861053 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:36:53.861029 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24\": container with ID starting with e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24 not found: ID does not exist" containerID="e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24" Nov 03 20:36:53.861158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.861059 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24"} err="failed to get container status \"e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24\": rpc error: code = NotFound desc = could not find container \"e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24\": container with ID starting with e7d8309b7741f2967d624bed9643f037f18691af1b0b35ade8883cac4031ee24 not found: ID does not exist" Nov 03 20:36:53.862656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.862630 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume\") pod \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " Nov 03 20:36:53.862817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.862799 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials\") pod \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " Nov 03 20:36:53.862887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.862857 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume\") pod \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " Nov 03 20:36:53.862950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.862893 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume\") pod \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " Nov 03 20:36:53.863042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863009 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume" pod "41d5ae05-5321-4a20-938b-c4dde8cd0ee5" (UID: "41d5ae05-5321-4a20-938b-c4dde8cd0ee5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:53.863131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863048 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6\") pod \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " Nov 03 20:36:53.863194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863133 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume\") pod \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " Nov 03 20:36:53.863252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863213 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials\") pod \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\" (UID: \"41d5ae05-5321-4a20-938b-c4dde8cd0ee5\") " Nov 03 20:36:53.863313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863282 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt\") pod \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\" (UID: \"1b760e2d-c334-46c9-b0d3-4453dce5e70d\") " Nov 03 20:36:53.863467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863441 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume" pod "41d5ae05-5321-4a20-938b-c4dde8cd0ee5" (UID: "41d5ae05-5321-4a20-938b-c4dde8cd0ee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:53.863891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.863871 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume" pod "1b760e2d-c334-46c9-b0d3-4453dce5e70d" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:53.864053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.864029 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume" pod "1b760e2d-c334-46c9-b0d3-4453dce5e70d" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:53.864441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.864423 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.864592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.864577 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.864711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.864696 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.864812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.864799 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b760e2d-c334-46c9-b0d3-4453dce5e70d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.866415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.866394 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6" pod "41d5ae05-5321-4a20-938b-c4dde8cd0ee5" (UID: "41d5ae05-5321-4a20-938b-c4dde8cd0ee5"). InnerVolumeSpecName "kube-api-access-clgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:53.866612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.866441 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials" pod "41d5ae05-5321-4a20-938b-c4dde8cd0ee5" (UID: "41d5ae05-5321-4a20-938b-c4dde8cd0ee5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:53.867162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.867141 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt" pod "1b760e2d-c334-46c9-b0d3-4453dce5e70d" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d"). InnerVolumeSpecName "kube-api-access-6rvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:53.867803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.867786 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials" pod "1b760e2d-c334-46c9-b0d3-4453dce5e70d" (UID: "1b760e2d-c334-46c9-b0d3-4453dce5e70d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:53.966386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.966355 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clgw6\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-kube-api-access-clgw6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.966386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.966384 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/41d5ae05-5321-4a20-938b-c4dde8cd0ee5-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.966602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.966395 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rvpt\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-kube-api-access-6rvpt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:53.966602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:53.966404 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b760e2d-c334-46c9-b0d3-4453dce5e70d-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:54.147820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:54.147778 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq"] Nov 03 20:36:54.152724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:54.152705 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-5589pblgq"] Nov 03 20:36:54.162691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:54.162659 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn"] Nov 03 20:36:54.166624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:54.166601 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-6bd6tlgtn"] Nov 03 20:36:54.590922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:54.590875 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b760e2d-c334-46c9-b0d3-4453dce5e70d" path="/var/lib/kubelet/pods/1b760e2d-c334-46c9-b0d3-4453dce5e70d/volumes" Nov 03 20:36:54.591797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:36:54.591769 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d5ae05-5321-4a20-938b-c4dde8cd0ee5" path="/var/lib/kubelet/pods/41d5ae05-5321-4a20-938b-c4dde8cd0ee5/volumes" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.536675 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q"] Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538059 5815 cpu_manager.go:414] "containerMap: removing container" podUID="41d5ae05-5321-4a20-938b-c4dde8cd0ee5" containerName="payload-processing" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538082 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d5ae05-5321-4a20-938b-c4dde8cd0ee5" containerName="payload-processing" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538104 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b0425a08-2b24-443d-8d3f-a182414d94ed" containerName="revoke-keys" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538112 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0425a08-2b24-443d-8d3f-a182414d94ed" containerName="revoke-keys" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538180 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1b760e2d-c334-46c9-b0d3-4453dce5e70d" containerName="payload-processing" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538188 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b760e2d-c334-46c9-b0d3-4453dce5e70d" containerName="payload-processing" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538382 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d5ae05-5321-4a20-938b-c4dde8cd0ee5" containerName="payload-processing" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538397 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0425a08-2b24-443d-8d3f-a182414d94ed" containerName="revoke-keys" Nov 03 20:37:03.544530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.538421 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b760e2d-c334-46c9-b0d3-4453dce5e70d" containerName="payload-processing" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.605532 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.642274 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q"] Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.669854 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"e2e-cmp-legacy-9b206f-provider-cred\"" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.669968 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-plugins-e2e-extmod-a-db5198\"" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.670044 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"kube-root-ca.crt\"" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.673011 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"routing-overlay\"" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.684480 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.684545 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbldb\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.684578 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.684671 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.726748 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"openshift-service-ca.crt\"" Nov 03 20:37:03.761545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.752635 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:03.807996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.802702 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.822448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.821547 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.822448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.821608 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbldb\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.822448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.821654 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.823074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.823043 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.823721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.823691 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.826390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.826363 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.841482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.841375 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbldb\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:03.884988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:03.884953 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:04.295209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:04.295184 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q"] Nov 03 20:37:04.296535 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:04.296489 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4bda77_152e_4ad9_83f8_2339830e9695.slice/crio-29bf30dc288e4e01ad346c56631dc8cce0253548497ca8e422b55549f97214e9 WatchSource:0}: Error finding container 29bf30dc288e4e01ad346c56631dc8cce0253548497ca8e422b55549f97214e9: Status 404 returned error can't find the container with id 29bf30dc288e4e01ad346c56631dc8cce0253548497ca8e422b55549f97214e9 Nov 03 20:37:04.610237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:04.610137 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" event={"ID":"0d4bda77-152e-4ad9-83f8-2339830e9695","Type":"ContainerStarted","Data":"32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e"} Nov 03 20:37:04.610743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:04.610361 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" event={"ID":"0d4bda77-152e-4ad9-83f8-2339830e9695","Type":"ContainerStarted","Data":"29bf30dc288e4e01ad346c56631dc8cce0253548497ca8e422b55549f97214e9"} Nov 03 20:37:04.610824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:04.610798 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:04.629683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:04.629636 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" podStartSLOduration=1.629624059 podStartE2EDuration="1.629624059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:04.629374508 +0000 UTC m=+2134.159720172" watchObservedRunningTime="2026-11-03 20:37:04.629624059 +0000 UTC m=+2134.159969764" Nov 03 20:37:04.932673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:04.931618 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:37:06.782717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:06.782675 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:06.948336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:06.948290 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:37:07.035810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:07.035019 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:07.071744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:07.070606 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:07.249202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:07.248380 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:07.249202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:07.248853 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:37:07.386846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:07.386192 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:10.022666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.022097 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n"] Nov 03 20:37:10.028551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.028484 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.035364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.035316 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"payload-processing-external-model-e2e--bb65a178-dockercfg-5zp5h\"" Nov 03 20:37:10.038388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.038363 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-502cc5aa7445\"" Nov 03 20:37:10.038615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.038598 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-d84007a46645\"" Nov 03 20:37:10.039066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.039048 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-371723960661\"" Nov 03 20:37:10.039415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.039397 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"openshift-service-ca.crt\"" Nov 03 20:37:10.040052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.040009 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"kube-root-ca.crt\"" Nov 03 20:37:10.040172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.040084 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"payload-processing-external-model-plugins-e2e-extmod-b-03c633\"" Nov 03 20:37:10.040235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.040192 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"routing-overlay\"" Nov 03 20:37:10.054158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.054118 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n"] Nov 03 20:37:10.171754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.171650 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.179809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.179776 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.195532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.180123 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mg5\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.210538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.205636 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.307704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.307606 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mg5\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.307704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.307672 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.307961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.307856 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.308024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.307965 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.309244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.308782 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.309244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.309196 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.312210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.312165 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.320713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.320687 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mg5\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.356919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.356899 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:10.654162 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:10.654127 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf62faa8_b63c_4e66_85fe_354238e0b6ef.slice/crio-0b4b4733c257eaf5a1e6edcd521267f399d1322774ad477d98328a69c3c31324 WatchSource:0}: Error finding container 0b4b4733c257eaf5a1e6edcd521267f399d1322774ad477d98328a69c3c31324: Status 404 returned error can't find the container with id 0b4b4733c257eaf5a1e6edcd521267f399d1322774ad477d98328a69c3c31324 Nov 03 20:37:10.656031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:10.656002 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n"] Nov 03 20:37:11.004215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:11.004183 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" event={"ID":"af62faa8-b63c-4e66-85fe-354238e0b6ef","Type":"ContainerStarted","Data":"be0129b04d03a40a425a2b3e49a37a932a4b6245bc540acf22e6226a4936a7f1"} Nov 03 20:37:11.004422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:11.004240 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" event={"ID":"af62faa8-b63c-4e66-85fe-354238e0b6ef","Type":"ContainerStarted","Data":"0b4b4733c257eaf5a1e6edcd521267f399d1322774ad477d98328a69c3c31324"} Nov 03 20:37:11.004422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:11.004277 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:11.025422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:11.025379 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" podStartSLOduration=2.025367881 podStartE2EDuration="2.025367881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:11.024348249 +0000 UTC m=+2140.554693923" watchObservedRunningTime="2026-11-03 20:37:11.025367881 +0000 UTC m=+2140.555713553" Nov 03 20:37:14.263680 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:14.259400 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:37:15.625012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:15.624985 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:37:16.873959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:16.873923 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw"] Nov 03 20:37:17.244535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.238390 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" containerID="cri-o://56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0" gracePeriod=30 Nov 03 20:37:17.700068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.699993 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2"] Nov 03 20:37:17.706536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.706493 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.711404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.711375 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:17.711628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.711432 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:37:17.738741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.738714 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2"] Nov 03 20:37:17.820430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.820400 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.820627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.820596 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7m2j\" (UniqueName: \"kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.820726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.820711 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.922599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.922557 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.923052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.922685 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7m2j\" (UniqueName: \"kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.923052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.922795 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.923497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.923467 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.925166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.925146 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:17.935183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:17.935161 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7m2j\" (UniqueName: \"kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j\") pod \"maas-api-revoke-sub-f0790b2f-g9cd2\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:18.038545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:18.035980 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:18.387657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:18.387622 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2"] Nov 03 20:37:18.388037 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:18.388017 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ef4b7e_a9d2_448a_9617_446d9fd1bbfa.slice/crio-1eddf15a32ce4c58bdc4870df48296df7335d75590198da55d13ba4d81460af7 WatchSource:0}: Error finding container 1eddf15a32ce4c58bdc4870df48296df7335d75590198da55d13ba4d81460af7: Status 404 returned error can't find the container with id 1eddf15a32ce4c58bdc4870df48296df7335d75590198da55d13ba4d81460af7 Nov 03 20:37:19.262651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:19.262619 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" event={"ID":"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa","Type":"ContainerStarted","Data":"21d3e2413184e1de9cda455eb9ba09c9e638a3a3b8658cbd21342faa959cb3ba"} Nov 03 20:37:19.263120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:19.262677 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" event={"ID":"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa","Type":"ContainerStarted","Data":"1eddf15a32ce4c58bdc4870df48296df7335d75590198da55d13ba4d81460af7"} Nov 03 20:37:19.285326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:19.285273 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" podStartSLOduration=2.285258842 podStartE2EDuration="2.285258842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:19.282049249 +0000 UTC m=+2148.812394921" watchObservedRunningTime="2026-11-03 20:37:19.285258842 +0000 UTC m=+2148.815604513" Nov 03 20:37:20.333921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:20.333572 5815 generic.go:356] "Generic (PLEG): container finished" podID="b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" containerID="21d3e2413184e1de9cda455eb9ba09c9e638a3a3b8658cbd21342faa959cb3ba" exitCode=0 Nov 03 20:37:20.334465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:20.334440 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" event={"ID":"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa","Type":"ContainerDied","Data":"21d3e2413184e1de9cda455eb9ba09c9e638a3a3b8658cbd21342faa959cb3ba"} Nov 03 20:37:21.550771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.550738 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:21.614824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.614750 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca\") pod \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " Nov 03 20:37:21.614969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.614828 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca\") pod \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " Nov 03 20:37:21.614969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.614894 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j\" (UniqueName: \"kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j\") pod \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\" (UID: \"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa\") " Nov 03 20:37:21.615284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.615250 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca" pod "b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" (UID: "b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:21.615895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.615853 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:21.617251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.617229 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca" pod "b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" (UID: "b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:21.617670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.617632 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j" pod "b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" (UID: "b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa"). InnerVolumeSpecName "kube-api-access-v7m2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:21.718111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.718077 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:21.718111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:21.718108 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7m2j\" (UniqueName: \"kubernetes.io/projected/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa-kube-api-access-v7m2j\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.018319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.018285 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:37:22.357318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.357294 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" Nov 03 20:37:22.357483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.357334 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2" event={"ID":"b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa","Type":"ContainerDied","Data":"1eddf15a32ce4c58bdc4870df48296df7335d75590198da55d13ba4d81460af7"} Nov 03 20:37:22.357483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.357389 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eddf15a32ce4c58bdc4870df48296df7335d75590198da55d13ba4d81460af7" Nov 03 20:37:22.576533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.576037 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:37:22.672898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.672868 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.673067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.672986 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.673385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.673352 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.673585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.673568 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.674265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.674239 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.674673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.674643 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:22.674796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.674779 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.675083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.675059 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:22.676019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.675986 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:22.676122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.676077 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.676236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.676217 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.676308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.676268 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket\") pod \"06f9fe17-453c-414a-b557-b2bae82e27b2\" (UID: \"06f9fe17-453c-414a-b557-b2bae82e27b2\") " Nov 03 20:37:22.677569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677051 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:22.677569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677054 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "kube-api-access-qn8k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:22.677569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677535 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06f9fe17-453c-414a-b557-b2bae82e27b2-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.677805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677738 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.677805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677760 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.677805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677775 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8k6\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-kube-api-access-qn8k6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.677805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677789 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.678027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.677936 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:22.678894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.678861 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:22.679731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.679707 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:22.680704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.680681 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy" pod "06f9fe17-453c-414a-b557-b2bae82e27b2" (UID: "06f9fe17-453c-414a-b557-b2bae82e27b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:22.779394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.779364 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.779394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.779391 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.779394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.779400 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:22.779698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:22.779408 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06f9fe17-453c-414a-b557-b2bae82e27b2-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:23.407685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.407642 5815 generic.go:356] "Generic (PLEG): container finished" podID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerID="56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0" exitCode=0 Nov 03 20:37:23.407915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.407794 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" event={"ID":"06f9fe17-453c-414a-b557-b2bae82e27b2","Type":"ContainerDied","Data":"56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0"} Nov 03 20:37:23.407915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.407836 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" event={"ID":"06f9fe17-453c-414a-b557-b2bae82e27b2","Type":"ContainerDied","Data":"043eb05d2e50abe38a27ef570126b1e68872fcfa7666cb714e210e12c06a726d"} Nov 03 20:37:23.407915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.407859 5815 scope.go:122] "RemoveContainer" containerID="56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0" Nov 03 20:37:23.408121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.408012 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw" Nov 03 20:37:23.425895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.425874 5815 scope.go:122] "RemoveContainer" containerID="56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0" Nov 03 20:37:23.426223 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:23.426203 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0\": container with ID starting with 56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0 not found: ID does not exist" containerID="56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0" Nov 03 20:37:23.426287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.426234 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0"} err="failed to get container status \"56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0\": rpc error: code = NotFound desc = could not find container \"56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0\": container with ID starting with 56eb14a587f5249dc2c0258d712a6fd23c1a7e1610fed5cc4d5a9338215e8df0 not found: ID does not exist" Nov 03 20:37:23.440346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.440315 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw"] Nov 03 20:37:23.445438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:23.445414 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-88msw"] Nov 03 20:37:24.589192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:24.589162 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" path="/var/lib/kubelet/pods/06f9fe17-453c-414a-b557-b2bae82e27b2/volumes" Nov 03 20:37:27.041298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.041261 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq"] Nov 03 20:37:27.042214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.042191 5815 cpu_manager.go:414] "containerMap: removing container" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" Nov 03 20:37:27.042214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.042212 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" Nov 03 20:37:27.042390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.042266 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" containerName="revoke-keys" Nov 03 20:37:27.042390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.042272 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" containerName="revoke-keys" Nov 03 20:37:27.042551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.042392 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" containerName="revoke-keys" Nov 03 20:37:27.042551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.042408 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f9fe17-453c-414a-b557-b2bae82e27b2" containerName="istio-proxy" Nov 03 20:37:27.047815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.047798 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq"] Nov 03 20:37:27.047935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.047879 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.137158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.137125 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsbt\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.137340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.137247 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.137340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.137320 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.137604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.137578 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.239148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.239113 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.239308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.239230 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsbt\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.239308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.239297 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.239397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.239352 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.240034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.240001 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.240034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.240026 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.241975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.241955 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.251613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.251588 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsbt\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.363035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.362960 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:27.675312 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:27.675282 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959f9758_18fa_4a15_87fe_eb450198a4d4.slice/crio-f83c1458583589b95649a176d333847d5b667c4557967295104a261e375f8dc9 WatchSource:0}: Error finding container f83c1458583589b95649a176d333847d5b667c4557967295104a261e375f8dc9: Status 404 returned error can't find the container with id f83c1458583589b95649a176d333847d5b667c4557967295104a261e375f8dc9 Nov 03 20:37:27.675450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:27.675388 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq"] Nov 03 20:37:28.634329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:28.634289 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" event={"ID":"959f9758-18fa-4a15-87fe-eb450198a4d4","Type":"ContainerStarted","Data":"f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290"} Nov 03 20:37:28.634838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:28.634359 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:28.634838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:28.634379 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" event={"ID":"959f9758-18fa-4a15-87fe-eb450198a4d4","Type":"ContainerStarted","Data":"f83c1458583589b95649a176d333847d5b667c4557967295104a261e375f8dc9"} Nov 03 20:37:28.655578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:28.655472 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" podStartSLOduration=2.655455274 podStartE2EDuration="2.655455274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:28.65478549 +0000 UTC m=+2158.185131163" watchObservedRunningTime="2026-11-03 20:37:28.655455274 +0000 UTC m=+2158.185800950" Nov 03 20:37:29.317570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:29.314540 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6"] Nov 03 20:37:30.772532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:30.772417 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc"] Nov 03 20:37:30.773093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:30.772681 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" containerID="cri-o://fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13" gracePeriod=30 Nov 03 20:37:33.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.910733 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq"] Nov 03 20:37:33.920812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.920776 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:33.926389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.926360 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:33.928329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.928020 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:37:33.929121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.928859 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq"] Nov 03 20:37:33.998052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.997940 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq"] Nov 03 20:37:33.998696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:33.998634 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" containerID="cri-o://8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7" gracePeriod=30 Nov 03 20:37:34.007164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.007138 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq"] Nov 03 20:37:34.007571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.007527 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" containerID="cri-o://69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94" gracePeriod=30 Nov 03 20:37:34.020284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.020259 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87"] Nov 03 20:37:34.020783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.020521 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" containerID="cri-o://fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043" gracePeriod=30 Nov 03 20:37:34.021254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.021214 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.021376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.021315 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qljlt\" (UniqueName: \"kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.021599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.021552 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.028794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.028773 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7"] Nov 03 20:37:34.029029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.028975 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" containerID="cri-o://f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d" gracePeriod=30 Nov 03 20:37:34.132819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.132778 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.133230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.133205 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qljlt\" (UniqueName: \"kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.133580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.133561 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.134657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.134635 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.139808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.139775 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.148050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.148027 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qljlt\" (UniqueName: \"kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt\") pod \"maas-api-revoke-sub-18f1b668-ql8sq\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.245490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.245468 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:34.315641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.315602 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" podUID="00611521-41ee-4155-a6c0-adfaf02190b8" containerName="payload-processing" containerID="cri-o://e2e0ceb5e8771151a8347484392f7632717aea8662e46631a8b27285f633fc91" gracePeriod=25 Nov 03 20:37:34.546544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.545048 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq"] Nov 03 20:37:34.550578 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:34.549995 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda52e4d9_934b_4775_992c_18c2db92df7a.slice/crio-70cff970f6c1b14e9b2a091808e372cd5472169a739f6ae9ca30858c417f7a71 WatchSource:0}: Error finding container 70cff970f6c1b14e9b2a091808e372cd5472169a739f6ae9ca30858c417f7a71: Status 404 returned error can't find the container with id 70cff970f6c1b14e9b2a091808e372cd5472169a739f6ae9ca30858c417f7a71 Nov 03 20:37:34.663481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.663447 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:37:34.663764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.663734 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:37:34.734325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.734284 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" event={"ID":"da52e4d9-934b-4775-992c-18c2db92df7a","Type":"ContainerStarted","Data":"70cff970f6c1b14e9b2a091808e372cd5472169a739f6ae9ca30858c417f7a71"} Nov 03 20:37:34.736299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.736274 5815 generic.go:356] "Generic (PLEG): container finished" podID="00611521-41ee-4155-a6c0-adfaf02190b8" containerID="e2e0ceb5e8771151a8347484392f7632717aea8662e46631a8b27285f633fc91" exitCode=0 Nov 03 20:37:34.736454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.736333 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" event={"ID":"00611521-41ee-4155-a6c0-adfaf02190b8","Type":"ContainerDied","Data":"e2e0ceb5e8771151a8347484392f7632717aea8662e46631a8b27285f633fc91"} Nov 03 20:37:34.797162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.797137 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:37:34.847856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.847817 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume\") pod \"00611521-41ee-4155-a6c0-adfaf02190b8\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " Nov 03 20:37:34.848038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.848015 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials\") pod \"00611521-41ee-4155-a6c0-adfaf02190b8\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " Nov 03 20:37:34.848147 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.848053 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm\") pod \"00611521-41ee-4155-a6c0-adfaf02190b8\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " Nov 03 20:37:34.848241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.848216 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume" pod "00611521-41ee-4155-a6c0-adfaf02190b8" (UID: "00611521-41ee-4155-a6c0-adfaf02190b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:34.848311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.848280 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume\") pod \"00611521-41ee-4155-a6c0-adfaf02190b8\" (UID: \"00611521-41ee-4155-a6c0-adfaf02190b8\") " Nov 03 20:37:34.848999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.848970 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:34.849616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.849586 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume" pod "00611521-41ee-4155-a6c0-adfaf02190b8" (UID: "00611521-41ee-4155-a6c0-adfaf02190b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:34.850908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.850880 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials" pod "00611521-41ee-4155-a6c0-adfaf02190b8" (UID: "00611521-41ee-4155-a6c0-adfaf02190b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:34.851479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.851453 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm" pod "00611521-41ee-4155-a6c0-adfaf02190b8" (UID: "00611521-41ee-4155-a6c0-adfaf02190b8"). InnerVolumeSpecName "kube-api-access-4jlpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:34.950213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.950180 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:34.950213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.950213 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jlpm\" (UniqueName: \"kubernetes.io/projected/00611521-41ee-4155-a6c0-adfaf02190b8-kube-api-access-4jlpm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:34.950654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:34.950227 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00611521-41ee-4155-a6c0-adfaf02190b8-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:35.749266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.749230 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" Nov 03 20:37:35.749266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.749242 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6" event={"ID":"00611521-41ee-4155-a6c0-adfaf02190b8","Type":"ContainerDied","Data":"1af219f482688c56b2eb2c63d451e43295f4fe357ceb4ac4a28ce1f8ea9eb4e9"} Nov 03 20:37:35.749644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.749329 5815 scope.go:122] "RemoveContainer" containerID="e2e0ceb5e8771151a8347484392f7632717aea8662e46631a8b27285f633fc91" Nov 03 20:37:35.752686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.751749 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" event={"ID":"da52e4d9-934b-4775-992c-18c2db92df7a","Type":"ContainerStarted","Data":"93beb12fc98f8f0414e524fcd25c54446078858068cdbfdad7207f13d7b61ef7"} Nov 03 20:37:35.777142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.777094 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" podStartSLOduration=2.777079878 podStartE2EDuration="2.777079878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:35.771972468 +0000 UTC m=+2165.302318142" watchObservedRunningTime="2026-11-03 20:37:35.777079878 +0000 UTC m=+2165.307425549" Nov 03 20:37:35.794219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.794186 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6"] Nov 03 20:37:35.801811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:35.801786 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-65ffj4qv6"] Nov 03 20:37:36.133593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.133569 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:37:36.175784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.175647 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.175784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.175734 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.175784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.175774 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.175784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.175803 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.176243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.175846 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.176243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.175954 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.176243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.176040 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.176243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.176105 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.176243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.176140 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs\") pod \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\" (UID: \"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf\") " Nov 03 20:37:36.177013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.176983 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.177158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.177114 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.177242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.177210 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.177355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.177330 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.177710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.177685 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.177868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.177846 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:36.184447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.184420 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.193689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.193537 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "kube-api-access-4g7dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:36.193689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.193604 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:36.193689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.193659 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token" pod "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" (UID: "3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336547 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336592 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336608 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336633 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336649 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7dn\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-kube-api-access-4g7dn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336667 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336681 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.336833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.336701 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.592589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.592537 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00611521-41ee-4155-a6c0-adfaf02190b8" path="/var/lib/kubelet/pods/00611521-41ee-4155-a6c0-adfaf02190b8/volumes" Nov 03 20:37:36.791248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.791216 5815 generic.go:356] "Generic (PLEG): container finished" podID="da52e4d9-934b-4775-992c-18c2db92df7a" containerID="93beb12fc98f8f0414e524fcd25c54446078858068cdbfdad7207f13d7b61ef7" exitCode=0 Nov 03 20:37:36.791429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.791304 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" event={"ID":"da52e4d9-934b-4775-992c-18c2db92df7a","Type":"ContainerDied","Data":"93beb12fc98f8f0414e524fcd25c54446078858068cdbfdad7207f13d7b61ef7"} Nov 03 20:37:36.793366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.793340 5815 generic.go:356] "Generic (PLEG): container finished" podID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerID="fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13" exitCode=0 Nov 03 20:37:36.793569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.793408 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" Nov 03 20:37:36.793569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.793427 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" event={"ID":"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf","Type":"ContainerDied","Data":"fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13"} Nov 03 20:37:36.793569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.793487 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc" event={"ID":"3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf","Type":"ContainerDied","Data":"fa534baa18d0529c210e38e01fa9422f02968137c220981eaa36ff8ea37b239a"} Nov 03 20:37:36.793569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.793565 5815 scope.go:122] "RemoveContainer" containerID="fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13" Nov 03 20:37:36.815824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.815806 5815 scope.go:122] "RemoveContainer" containerID="fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13" Nov 03 20:37:36.816655 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:36.816589 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13\": container with ID starting with fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13 not found: ID does not exist" containerID="fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13" Nov 03 20:37:36.816655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.816628 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13"} err="failed to get container status \"fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13\": rpc error: code = NotFound desc = could not find container \"fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13\": container with ID starting with fb7e4f25a99c5f2633c63e45bfd2a000738b3679103fa8694f2663dd2cae4d13 not found: ID does not exist" Nov 03 20:37:36.834665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.834637 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc"] Nov 03 20:37:36.839393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.839370 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-nj4bc"] Nov 03 20:37:36.860430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.860376 5815 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:37:36.860548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:36.860429 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:37:37.995754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:37.995726 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:38.102433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.102337 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca\") pod \"da52e4d9-934b-4775-992c-18c2db92df7a\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " Nov 03 20:37:38.103130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.103106 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt\" (UniqueName: \"kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt\") pod \"da52e4d9-934b-4775-992c-18c2db92df7a\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " Nov 03 20:37:38.103846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.103826 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca\") pod \"da52e4d9-934b-4775-992c-18c2db92df7a\" (UID: \"da52e4d9-934b-4775-992c-18c2db92df7a\") " Nov 03 20:37:38.105323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.104372 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca" pod "da52e4d9-934b-4775-992c-18c2db92df7a" (UID: "da52e4d9-934b-4775-992c-18c2db92df7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:38.107309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.107286 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt" pod "da52e4d9-934b-4775-992c-18c2db92df7a" (UID: "da52e4d9-934b-4775-992c-18c2db92df7a"). InnerVolumeSpecName "kube-api-access-qljlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:38.107527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.107488 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca" pod "da52e4d9-934b-4775-992c-18c2db92df7a" (UID: "da52e4d9-934b-4775-992c-18c2db92df7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:38.212603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.209211 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:38.212603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.209253 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qljlt\" (UniqueName: \"kubernetes.io/projected/da52e4d9-934b-4775-992c-18c2db92df7a-kube-api-access-qljlt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:38.212603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.209270 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da52e4d9-934b-4775-992c-18c2db92df7a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:38.592433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.592355 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" path="/var/lib/kubelet/pods/3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf/volumes" Nov 03 20:37:38.821291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.821262 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" Nov 03 20:37:38.821471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.821294 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq" event={"ID":"da52e4d9-934b-4775-992c-18c2db92df7a","Type":"ContainerDied","Data":"70cff970f6c1b14e9b2a091808e372cd5472169a739f6ae9ca30858c417f7a71"} Nov 03 20:37:38.821471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:38.821342 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70cff970f6c1b14e9b2a091808e372cd5472169a739f6ae9ca30858c417f7a71" Nov 03 20:37:39.374388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.374361 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:37:39.441922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441522 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.441922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441616 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.441922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441678 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.441922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441714 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.441922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441785 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.441922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441790 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.442371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.441937 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.442371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.442017 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.442371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.442135 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.442371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.442177 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert\") pod \"7aa8313c-5794-49fd-8e97-caae0f755923\" (UID: \"7aa8313c-5794-49fd-8e97-caae0f755923\") " Nov 03 20:37:39.443436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.442825 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.443436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.443205 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.443436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.443228 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.443436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.443408 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.443753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.443663 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:39.443753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.443730 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.445948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.445923 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "kube-api-access-tdhrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.446837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.446808 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:39.447788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.447759 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.448083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.448047 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy" pod "7aa8313c-5794-49fd-8e97-caae0f755923" (UID: "7aa8313c-5794-49fd-8e97-caae0f755923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.509786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.509759 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:37:39.520930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.520909 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:37:39.523008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.522986 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:37:39.544606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544582 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.544730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544631 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.544730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544654 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.544730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544673 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.544730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544706 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.544961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544794 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.544961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544833 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.544961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544862 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.545106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.544976 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.545106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.545081 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.545178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.545110 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:39.545291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.545269 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.545359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.545343 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data\") pod \"6a461d60-810b-4cdb-9242-c46208bccb7e\" (UID: \"6a461d60-810b-4cdb-9242-c46208bccb7e\") " Nov 03 20:37:39.546296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546275 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546304 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546319 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546333 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546347 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aa8313c-5794-49fd-8e97-caae0f755923-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546361 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aa8313c-5794-49fd-8e97-caae0f755923-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546375 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdhrp\" (UniqueName: \"kubernetes.io/projected/7aa8313c-5794-49fd-8e97-caae0f755923-kube-api-access-tdhrp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546390 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546404 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546419 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a461d60-810b-4cdb-9242-c46208bccb7e-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546433 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aa8313c-5794-49fd-8e97-caae0f755923-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.546797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.546684 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.547030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.547004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.547553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.547529 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.547651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.547589 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:39.549026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.549004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597" pod "6a461d60-810b-4cdb-9242-c46208bccb7e" (UID: "6a461d60-810b-4cdb-9242-c46208bccb7e"). InnerVolumeSpecName "kube-api-access-zh597". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.647442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647342 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.647646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647490 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.647646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647593 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.647646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647635 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.647855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647655 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.647855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647741 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.647855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647776 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.647855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647823 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.647855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647848 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.648125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647946 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.648125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.647982 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.648125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648007 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.648125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648047 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.648125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648106 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.648409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648141 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.648409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648178 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv\") pod \"abbaa43a-6433-408b-9c35-87a7a44a6169\" (UID: \"abbaa43a-6433-408b-9c35-87a7a44a6169\") " Nov 03 20:37:39.648409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.648409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648258 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data\") pod \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\" (UID: \"d721d008-e6c7-4c0c-86c5-f28e73bb6661\") " Nov 03 20:37:39.648654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648542 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:39.648768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648740 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.648959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648847 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.649051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.648974 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.649313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649291 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:39.649313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649304 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649293 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649327 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649343 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649356 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649369 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649383 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh597\" (UniqueName: \"kubernetes.io/projected/6a461d60-810b-4cdb-9242-c46208bccb7e-kube-api-access-zh597\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649397 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a461d60-810b-4cdb-9242-c46208bccb7e-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649409 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.649483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649423 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.650081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649500 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.650081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649740 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.650081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.649912 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.651248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.651203 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.652872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.652847 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "kube-api-access-gswmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.653290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.653060 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.653290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.653076 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.653290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.653169 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.653290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.653174 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:39.653290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.653265 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo" pod "abbaa43a-6433-408b-9c35-87a7a44a6169" (UID: "abbaa43a-6433-408b-9c35-87a7a44a6169"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:39.654420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.654401 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "kube-api-access-57ksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:39.654865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.654849 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo" pod "d721d008-e6c7-4c0c-86c5-f28e73bb6661" (UID: "d721d008-e6c7-4c0c-86c5-f28e73bb6661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:39.686247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.686225 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:37:39.756531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.752980 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.765076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.764766 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbaa43a-6433-408b-9c35-87a7a44a6169-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.765596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.765575 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.765746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.765732 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.765863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.765852 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.765971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.765959 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766055 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gswmv\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-kube-api-access-gswmv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766131 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766146 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766159 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766174 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57ksj\" (UniqueName: \"kubernetes.io/projected/d721d008-e6c7-4c0c-86c5-f28e73bb6661-kube-api-access-57ksj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766189 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbaa43a-6433-408b-9c35-87a7a44a6169-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766207 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbaa43a-6433-408b-9c35-87a7a44a6169-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.766238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.766220 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d721d008-e6c7-4c0c-86c5-f28e73bb6661-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:39.801492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.801454 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh"] Nov 03 20:37:39.838650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.837622 5815 generic.go:356] "Generic (PLEG): container finished" podID="7aa8313c-5794-49fd-8e97-caae0f755923" containerID="8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7" exitCode=0 Nov 03 20:37:39.838650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.837956 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" Nov 03 20:37:39.838650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.838618 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" event={"ID":"7aa8313c-5794-49fd-8e97-caae0f755923","Type":"ContainerDied","Data":"8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7"} Nov 03 20:37:39.838968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.838680 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq" event={"ID":"7aa8313c-5794-49fd-8e97-caae0f755923","Type":"ContainerDied","Data":"2422bffea716c64fe7c1a026ed9b3482e14b9d01483a22b7631ed9dfc8f4eec3"} Nov 03 20:37:39.838968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.838712 5815 scope.go:122] "RemoveContainer" containerID="8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7" Nov 03 20:37:39.844563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.841615 5815 generic.go:356] "Generic (PLEG): container finished" podID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerID="69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94" exitCode=0 Nov 03 20:37:39.844563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.841680 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" event={"ID":"6a461d60-810b-4cdb-9242-c46208bccb7e","Type":"ContainerDied","Data":"69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94"} Nov 03 20:37:39.844563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.841713 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" event={"ID":"6a461d60-810b-4cdb-9242-c46208bccb7e","Type":"ContainerDied","Data":"88e2c949c8ce529a84f0e1e603c2a9065a59b04084782e06292af75440a83cd6"} Nov 03 20:37:39.844563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.841788 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq" Nov 03 20:37:39.850558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.848216 5815 generic.go:356] "Generic (PLEG): container finished" podID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerID="f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d" exitCode=0 Nov 03 20:37:39.850558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.848347 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" event={"ID":"abbaa43a-6433-408b-9c35-87a7a44a6169","Type":"ContainerDied","Data":"f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d"} Nov 03 20:37:39.850558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.848392 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" event={"ID":"abbaa43a-6433-408b-9c35-87a7a44a6169","Type":"ContainerDied","Data":"f814a2457fec7f04f18bce8fc2284f8b85f4dfb8a9950160372f47a542b10488"} Nov 03 20:37:39.850558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.848484 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7" Nov 03 20:37:39.855882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.855815 5815 generic.go:356] "Generic (PLEG): container finished" podID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerID="fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043" exitCode=0 Nov 03 20:37:39.856012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.855972 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" event={"ID":"d721d008-e6c7-4c0c-86c5-f28e73bb6661","Type":"ContainerDied","Data":"fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043"} Nov 03 20:37:39.856012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.856000 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" event={"ID":"d721d008-e6c7-4c0c-86c5-f28e73bb6661","Type":"ContainerDied","Data":"d92a9ae85f9832365ae4f676298a54d7b9da2d8365b90c09befdcdd4674f64c6"} Nov 03 20:37:39.856143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.856057 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87" Nov 03 20:37:39.857679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.857659 5815 scope.go:122] "RemoveContainer" containerID="0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4" Nov 03 20:37:39.889227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.888499 5815 scope.go:122] "RemoveContainer" containerID="8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7" Nov 03 20:37:39.889369 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:39.889307 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7\": container with ID starting with 8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7 not found: ID does not exist" containerID="8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7" Nov 03 20:37:39.889369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.889346 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7"} err="failed to get container status \"8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7\": rpc error: code = NotFound desc = could not find container \"8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7\": container with ID starting with 8431791f6dbf1cb97221e775988bf4e82e637f3a29ddfb8f89c49ce9630091b7 not found: ID does not exist" Nov 03 20:37:39.889536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.889374 5815 scope.go:122] "RemoveContainer" containerID="0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4" Nov 03 20:37:39.889768 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:39.889738 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4\": container with ID starting with 0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4 not found: ID does not exist" containerID="0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4" Nov 03 20:37:39.889855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.889774 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4"} err="failed to get container status \"0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4\": rpc error: code = NotFound desc = could not find container \"0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4\": container with ID starting with 0965a0545beed0cd98850d9a334e077471771ee6266fa365c379f6bbe3bab2d4 not found: ID does not exist" Nov 03 20:37:39.889855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.889798 5815 scope.go:122] "RemoveContainer" containerID="69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94" Nov 03 20:37:39.898536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.898466 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq"] Nov 03 20:37:39.903098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.903077 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-g58hq"] Nov 03 20:37:39.924807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.921661 5815 scope.go:122] "RemoveContainer" containerID="69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94" Nov 03 20:37:39.928593 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:39.926076 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94\": container with ID starting with 69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94 not found: ID does not exist" containerID="69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94" Nov 03 20:37:39.928593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.926121 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94"} err="failed to get container status \"69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94\": rpc error: code = NotFound desc = could not find container \"69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94\": container with ID starting with 69107e1f5d48366cb17d33cedd43195c4529db5da6b765fe6f121787da5bad94 not found: ID does not exist" Nov 03 20:37:39.928593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.926143 5815 scope.go:122] "RemoveContainer" containerID="f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d" Nov 03 20:37:39.929683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.929483 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq"] Nov 03 20:37:39.936829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.936802 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-ncqdq"] Nov 03 20:37:39.942703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.942676 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7"] Nov 03 20:37:39.943317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.943293 5815 scope.go:122] "RemoveContainer" containerID="f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d" Nov 03 20:37:39.943726 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:39.943702 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d\": container with ID starting with f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d not found: ID does not exist" containerID="f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d" Nov 03 20:37:39.943807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.943738 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d"} err="failed to get container status \"f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d\": rpc error: code = NotFound desc = could not find container \"f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d\": container with ID starting with f2dde6d8a4644b405e778c36eea62ffbf7cbc3c898cb045b89439f7d49541d3d not found: ID does not exist" Nov 03 20:37:39.943807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.943765 5815 scope.go:122] "RemoveContainer" containerID="fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043" Nov 03 20:37:39.947869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.947848 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-92mq7"] Nov 03 20:37:39.957896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.957755 5815 scope.go:122] "RemoveContainer" containerID="fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043" Nov 03 20:37:39.958316 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:39.958290 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043\": container with ID starting with fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043 not found: ID does not exist" containerID="fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043" Nov 03 20:37:39.958448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.958423 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043"} err="failed to get container status \"fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043\": rpc error: code = NotFound desc = could not find container \"fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043\": container with ID starting with fcca764d09e2a333ded6df86e8b89481ab454ae416a320b7bf29094479075043 not found: ID does not exist" Nov 03 20:37:39.960008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.959994 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87"] Nov 03 20:37:39.964142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:39.964122 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-5pv87"] Nov 03 20:37:40.589941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:40.589877 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" path="/var/lib/kubelet/pods/6a461d60-810b-4cdb-9242-c46208bccb7e/volumes" Nov 03 20:37:40.590603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:40.590580 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" path="/var/lib/kubelet/pods/7aa8313c-5794-49fd-8e97-caae0f755923/volumes" Nov 03 20:37:40.591129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:40.591111 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" path="/var/lib/kubelet/pods/abbaa43a-6433-408b-9c35-87a7a44a6169/volumes" Nov 03 20:37:40.591962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:40.591941 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" path="/var/lib/kubelet/pods/d721d008-e6c7-4c0c-86c5-f28e73bb6661/volumes" Nov 03 20:37:41.720466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:41.720433 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x"] Nov 03 20:37:41.720863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:41.720706 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" containerID="cri-o://cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608" gracePeriod=30 Nov 03 20:37:43.414807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.414768 5815 scope.go:122] "RemoveContainer" containerID="4259941579d545e3301ae95ca0618d81c651bd88000b8a4ede77b664de29dc14" Nov 03 20:37:43.428543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.428474 5815 scope.go:122] "RemoveContainer" containerID="3991d407927b63b89f66544d58d54db287f7786bd3407ffd2a83feee58bcf387" Nov 03 20:37:43.445127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.445096 5815 scope.go:122] "RemoveContainer" containerID="f55155f4b7aa8759762fd58d452fae84c48672add53d8770585e0cb97598c790" Nov 03 20:37:43.459044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.459022 5815 scope.go:122] "RemoveContainer" containerID="c6939f296cefada88ddce1c7f8a0e00bafff52984130ca32ea5d4f3900707f9d" Nov 03 20:37:43.472586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.471793 5815 scope.go:122] "RemoveContainer" containerID="e27a80245651b16c252a93e1a53cd3e5cdf0c427b7cf5b46600bff310f7de530" Nov 03 20:37:43.483535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.483495 5815 scope.go:122] "RemoveContainer" containerID="f102dceb58688be1ad1e03cf990e2c3c5646f5b7ad16ba2975a806d95a1dc557" Nov 03 20:37:43.495025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.495006 5815 scope.go:122] "RemoveContainer" containerID="3427088eddd691624fa192567be98558f92dffc624bc95d1920780b4ff3b1efc" Nov 03 20:37:43.505330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:43.505313 5815 scope.go:122] "RemoveContainer" containerID="18b5a48140320e0f676b8d6773775704fcbce91a5e86c590cb63d3a61e14859d" Nov 03 20:37:43.854361 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:43.854330 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:37:44.823301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:44.821009 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" podUID="335b171b-9980-4593-8806-1b95b1ab7fd3" containerName="payload-processing" containerID="cri-o://a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8" gracePeriod=25 Nov 03 20:37:45.059400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.041544 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:45.095606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.089296 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:37:45.143657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.142608 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:45.337707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.233627 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:45.538895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.537426 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.865528 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf"] Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.875175 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.875201 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.875234 5815 cpu_manager.go:414] "containerMap: removing container" podUID="da52e4d9-934b-4775-992c-18c2db92df7a" containerName="revoke-keys" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.875247 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da52e4d9-934b-4775-992c-18c2db92df7a" containerName="revoke-keys" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.875267 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.875276 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883195 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883219 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883301 5815 cpu_manager.go:414] "containerMap: removing container" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883310 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883347 5815 cpu_manager.go:414] "containerMap: removing container" podUID="00611521-41ee-4155-a6c0-adfaf02190b8" containerName="payload-processing" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883356 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00611521-41ee-4155-a6c0-adfaf02190b8" containerName="payload-processing" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883402 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.883410 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.884976 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.884995 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.890216 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbaa43a-6433-408b-9c35-87a7a44a6169" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.890244 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="00611521-41ee-4155-a6c0-adfaf02190b8" containerName="payload-processing" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.890465 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a461d60-810b-4cdb-9242-c46208bccb7e" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.890500 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d721d008-e6c7-4c0c-86c5-f28e73bb6661" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.891579 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.891847 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="da52e4d9-934b-4775-992c-18c2db92df7a" containerName="revoke-keys" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.891869 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8bdf54-ebd5-4b0b-ae0d-fc86f5bb89bf" containerName="istio-proxy" Nov 03 20:37:45.910022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.891881 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa8313c-5794-49fd-8e97-caae0f755923" containerName="istio-proxy" Nov 03 20:37:45.914001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.913968 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:45.931404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.931365 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-plugins-e2e-extmod-a-37aa95\"" Nov 03 20:37:45.932071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.932023 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-e2e--0dce6837-dockercfg-c69fq\"" Nov 03 20:37:45.932477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.932455 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-5797404fedc5\"" Nov 03 20:37:45.933031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.933006 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"kube-root-ca.crt\"" Nov 03 20:37:45.967153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.967121 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"routing-overlay\"" Nov 03 20:37:45.992660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.980572 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:37:45.993018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:45.982655 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"openshift-service-ca.crt\"" Nov 03 20:37:46.007830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.007790 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:37:46.034539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.010695 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf"] Nov 03 20:37:46.107535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.087154 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.107535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.087278 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.107535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.087501 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmgg\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.107535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.087673 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.107535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.103580 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6"] Nov 03 20:37:46.150234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.141339 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.150234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.142106 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6"] Nov 03 20:37:46.151080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.151054 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:37:46.151373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.151356 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:46.197706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.197664 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmgg\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.201718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.201634 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.202008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.201994 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.202128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.202118 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.202781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.202735 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.203609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.203592 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.204278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.204264 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrsk6\" (UniqueName: \"kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.204542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.204148 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.204658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.203454 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.207904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.207855 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.213336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.213311 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmgg\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.306461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.306422 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrsk6\" (UniqueName: \"kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.306748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.306637 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.306748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.306725 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.307903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.307876 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.311453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.311424 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.320814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.320757 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrsk6\" (UniqueName: \"kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6\") pod \"maas-api-revoke-sub-3200e506-kv8f6\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.343338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.342316 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:37:46.363973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.363946 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:46.407946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.407899 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj\") pod \"335b171b-9980-4593-8806-1b95b1ab7fd3\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " Nov 03 20:37:46.408309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.408287 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume\") pod \"335b171b-9980-4593-8806-1b95b1ab7fd3\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " Nov 03 20:37:46.408462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.408447 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume\") pod \"335b171b-9980-4593-8806-1b95b1ab7fd3\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " Nov 03 20:37:46.408641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.408624 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials\") pod \"335b171b-9980-4593-8806-1b95b1ab7fd3\" (UID: \"335b171b-9980-4593-8806-1b95b1ab7fd3\") " Nov 03 20:37:46.408768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.408738 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume" pod "335b171b-9980-4593-8806-1b95b1ab7fd3" (UID: "335b171b-9980-4593-8806-1b95b1ab7fd3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:46.409215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.409189 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume" pod "335b171b-9980-4593-8806-1b95b1ab7fd3" (UID: "335b171b-9980-4593-8806-1b95b1ab7fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:46.409718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.409698 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:46.409849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.409836 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/335b171b-9980-4593-8806-1b95b1ab7fd3-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:46.412621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.412582 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj" pod "335b171b-9980-4593-8806-1b95b1ab7fd3" (UID: "335b171b-9980-4593-8806-1b95b1ab7fd3"). InnerVolumeSpecName "kube-api-access-p5sgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:46.413146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.413123 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials" pod "335b171b-9980-4593-8806-1b95b1ab7fd3" (UID: "335b171b-9980-4593-8806-1b95b1ab7fd3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:46.478733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.473929 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:46.512499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.512184 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sgj\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-kube-api-access-p5sgj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:46.512499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.512222 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/335b171b-9980-4593-8806-1b95b1ab7fd3-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:46.749818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.749778 5815 generic.go:356] "Generic (PLEG): container finished" podID="335b171b-9980-4593-8806-1b95b1ab7fd3" containerID="a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8" exitCode=0 Nov 03 20:37:46.749944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.749880 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" event={"ID":"335b171b-9980-4593-8806-1b95b1ab7fd3","Type":"ContainerDied","Data":"a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8"} Nov 03 20:37:46.749944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.749936 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" event={"ID":"335b171b-9980-4593-8806-1b95b1ab7fd3","Type":"ContainerDied","Data":"af4e7f44c8164236cd88e564be18f870d14db26d177b5ef72f9eaa2982c79b55"} Nov 03 20:37:46.750053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.749970 5815 scope.go:122] "RemoveContainer" containerID="a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8" Nov 03 20:37:46.750186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.750160 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh" Nov 03 20:37:46.828280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.828251 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh"] Nov 03 20:37:46.834321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.833940 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6cb5dxdhh"] Nov 03 20:37:46.847888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.847781 5815 scope.go:122] "RemoveContainer" containerID="a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8" Nov 03 20:37:46.852057 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:46.852019 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8\": container with ID starting with a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8 not found: ID does not exist" containerID="a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8" Nov 03 20:37:46.852190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.852069 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8"} err="failed to get container status \"a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8\": rpc error: code = NotFound desc = could not find container \"a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8\": container with ID starting with a0671df60b206c5d9f563c92e9cc409ff818cf4b069ac0ccb25803dc3dee78e8 not found: ID does not exist" Nov 03 20:37:46.918362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:46.918315 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf"] Nov 03 20:37:46.932960 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:46.932933 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d32ff7_7f4b_43fb_8082_fdc26b2e36eb.slice/crio-0ef092ccde4fa19360260cfe049b649f78c6d0e375fe217d4788b1308a9fe034 WatchSource:0}: Error finding container 0ef092ccde4fa19360260cfe049b649f78c6d0e375fe217d4788b1308a9fe034: Status 404 returned error can't find the container with id 0ef092ccde4fa19360260cfe049b649f78c6d0e375fe217d4788b1308a9fe034 Nov 03 20:37:47.121113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.120318 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6"] Nov 03 20:37:47.126145 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:47.126113 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aef707_29ed_4249_9d05_11c637aefdfb.slice/crio-245adb8a4afbdcc39101d995e1aa865a5999aa05f1e8f9ae0c515e0059f4aa2f WatchSource:0}: Error finding container 245adb8a4afbdcc39101d995e1aa865a5999aa05f1e8f9ae0c515e0059f4aa2f: Status 404 returned error can't find the container with id 245adb8a4afbdcc39101d995e1aa865a5999aa05f1e8f9ae0c515e0059f4aa2f Nov 03 20:37:47.462736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.462709 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589171 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589230 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589285 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589330 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589369 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589424 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589479 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589621 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.592532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.589671 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy\") pod \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\" (UID: \"9dbf43e9-9dec-44d3-80b3-ef123df87ebb\") " Nov 03 20:37:47.596876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.596844 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:47.597214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.597182 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.597320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.597192 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.597884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.597774 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.598780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.598749 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.599490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.599466 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.602204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.602168 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:47.604423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.604387 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:47.604683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.604663 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg" pod "9dbf43e9-9dec-44d3-80b3-ef123df87ebb" (UID: "9dbf43e9-9dec-44d3-80b3-ef123df87ebb"). InnerVolumeSpecName "kube-api-access-splzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694285 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694322 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694339 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694353 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694367 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694382 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694395 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694409 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splzg\" (UniqueName: \"kubernetes.io/projected/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-kube-api-access-splzg\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.695593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.694423 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dbf43e9-9dec-44d3-80b3-ef123df87ebb-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.762693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.762659 5815 generic.go:356] "Generic (PLEG): container finished" podID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerID="cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608" exitCode=0 Nov 03 20:37:47.762879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.762719 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" Nov 03 20:37:47.762879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.762745 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" event={"ID":"9dbf43e9-9dec-44d3-80b3-ef123df87ebb","Type":"ContainerDied","Data":"cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608"} Nov 03 20:37:47.762879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.762808 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x" event={"ID":"9dbf43e9-9dec-44d3-80b3-ef123df87ebb","Type":"ContainerDied","Data":"39c91d95a07facda4aaac7e599e7d68854b4ec77a43186867ae36132607e9be2"} Nov 03 20:37:47.762879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.762838 5815 scope.go:122] "RemoveContainer" containerID="cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608" Nov 03 20:37:47.780364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.780247 5815 scope.go:122] "RemoveContainer" containerID="cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608" Nov 03 20:37:47.790333 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:47.789784 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608\": container with ID starting with cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608 not found: ID does not exist" containerID="cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608" Nov 03 20:37:47.790333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.789826 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608"} err="failed to get container status \"cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608\": rpc error: code = NotFound desc = could not find container \"cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608\": container with ID starting with cb638ca2adf73282dec995cf6042e2d9bdfc580525f5937cf192f1d7daeb3608 not found: ID does not exist" Nov 03 20:37:47.796530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.791438 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" event={"ID":"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb","Type":"ContainerStarted","Data":"695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270"} Nov 03 20:37:47.796530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.791493 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" event={"ID":"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb","Type":"ContainerStarted","Data":"0ef092ccde4fa19360260cfe049b649f78c6d0e375fe217d4788b1308a9fe034"} Nov 03 20:37:47.799738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.799711 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:37:47.799971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.799940 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" event={"ID":"e3aef707-29ed-4249-9d05-11c637aefdfb","Type":"ContainerStarted","Data":"1b9ae43530da21a854bde59db1c9faf369a1986bd4579b0010f77dbf36236360"} Nov 03 20:37:47.800070 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.800002 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" event={"ID":"e3aef707-29ed-4249-9d05-11c637aefdfb","Type":"ContainerStarted","Data":"245adb8a4afbdcc39101d995e1aa865a5999aa05f1e8f9ae0c515e0059f4aa2f"} Nov 03 20:37:47.821558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.821516 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" podStartSLOduration=2.821491313 podStartE2EDuration="2.821491313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:47.819651426 +0000 UTC m=+2177.349997099" watchObservedRunningTime="2026-11-03 20:37:47.821491313 +0000 UTC m=+2177.351836985" Nov 03 20:37:47.837236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.837213 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x"] Nov 03 20:37:47.841569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.841546 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-dmc6x"] Nov 03 20:37:47.854209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:47.854158 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" podStartSLOduration=2.854146299 podStartE2EDuration="2.854146299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:47.853063628 +0000 UTC m=+2177.383409300" watchObservedRunningTime="2026-11-03 20:37:47.854146299 +0000 UTC m=+2177.384491971" Nov 03 20:37:48.602936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:48.602899 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335b171b-9980-4593-8806-1b95b1ab7fd3" path="/var/lib/kubelet/pods/335b171b-9980-4593-8806-1b95b1ab7fd3/volumes" Nov 03 20:37:48.604485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:48.603783 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" path="/var/lib/kubelet/pods/9dbf43e9-9dec-44d3-80b3-ef123df87ebb/volumes" Nov 03 20:37:48.869585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:48.866395 5815 generic.go:356] "Generic (PLEG): container finished" podID="e3aef707-29ed-4249-9d05-11c637aefdfb" containerID="1b9ae43530da21a854bde59db1c9faf369a1986bd4579b0010f77dbf36236360" exitCode=0 Nov 03 20:37:48.869585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:48.866683 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" event={"ID":"e3aef707-29ed-4249-9d05-11c637aefdfb","Type":"ContainerDied","Data":"1b9ae43530da21a854bde59db1c9faf369a1986bd4579b0010f77dbf36236360"} Nov 03 20:37:49.626430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.520570 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw"] Nov 03 20:37:49.628412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.628380 5815 cpu_manager.go:414] "containerMap: removing container" podUID="335b171b-9980-4593-8806-1b95b1ab7fd3" containerName="payload-processing" Nov 03 20:37:49.628606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.628588 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335b171b-9980-4593-8806-1b95b1ab7fd3" containerName="payload-processing" Nov 03 20:37:49.628710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.628700 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" Nov 03 20:37:49.628789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.628778 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" Nov 03 20:37:49.629126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.629107 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="335b171b-9980-4593-8806-1b95b1ab7fd3" containerName="payload-processing" Nov 03 20:37:49.629251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.629238 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbf43e9-9dec-44d3-80b3-ef123df87ebb" containerName="istio-proxy" Nov 03 20:37:49.635118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.635099 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw"] Nov 03 20:37:49.636345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.636305 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.743713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.743677 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.743914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.743764 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.743914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.743789 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.743914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.743870 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9zr\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.844753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.844715 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.844753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.844760 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9zr\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.845027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.844907 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.845117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.845081 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.845680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.845646 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.845979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.845956 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.848451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.848420 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.876693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.876620 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9zr\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:49.961821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:49.961793 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:50.408066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.407675 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:50.524870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.524782 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca\") pod \"e3aef707-29ed-4249-9d05-11c637aefdfb\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " Nov 03 20:37:50.525398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.525375 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca\") pod \"e3aef707-29ed-4249-9d05-11c637aefdfb\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " Nov 03 20:37:50.525800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.525779 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca" pod "e3aef707-29ed-4249-9d05-11c637aefdfb" (UID: "e3aef707-29ed-4249-9d05-11c637aefdfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:50.525954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.525791 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6\" (UniqueName: \"kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6\") pod \"e3aef707-29ed-4249-9d05-11c637aefdfb\" (UID: \"e3aef707-29ed-4249-9d05-11c637aefdfb\") " Nov 03 20:37:50.526743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.526721 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:50.527731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.527698 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca" pod "e3aef707-29ed-4249-9d05-11c637aefdfb" (UID: "e3aef707-29ed-4249-9d05-11c637aefdfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:50.528756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.528736 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6" pod "e3aef707-29ed-4249-9d05-11c637aefdfb" (UID: "e3aef707-29ed-4249-9d05-11c637aefdfb"). InnerVolumeSpecName "kube-api-access-wrsk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:50.624790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.624488 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk"] Nov 03 20:37:50.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.625710 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e3aef707-29ed-4249-9d05-11c637aefdfb" containerName="revoke-keys" Nov 03 20:37:50.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.625734 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3aef707-29ed-4249-9d05-11c637aefdfb" containerName="revoke-keys" Nov 03 20:37:50.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.625944 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3aef707-29ed-4249-9d05-11c637aefdfb" containerName="revoke-keys" Nov 03 20:37:50.629371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.629308 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrsk6\" (UniqueName: \"kubernetes.io/projected/e3aef707-29ed-4249-9d05-11c637aefdfb-kube-api-access-wrsk6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:50.629371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.629335 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3aef707-29ed-4249-9d05-11c637aefdfb-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:50.632673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.632647 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw"] Nov 03 20:37:50.632769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.632697 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk"] Nov 03 20:37:50.633032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.633015 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.732030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.731842 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwdn\" (UniqueName: \"kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.732030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.731957 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.732407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.732381 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.834256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.834216 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwdn\" (UniqueName: \"kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.835238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.834324 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.835238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.835035 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.835238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.835190 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.841105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.837702 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.847394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.847367 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwdn\" (UniqueName: \"kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn\") pod \"maas-api-revoke-sub-f23d617d-kwwrk\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:50.925413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.925074 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" event={"ID":"028ec67a-77a0-40aa-8d3b-1abc794287e7","Type":"ContainerStarted","Data":"8f998d13e5cb55185178e6d6393190683418e6bdb0a0400c8ec554c2c8a5ea8e"} Nov 03 20:37:50.925413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.925175 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" event={"ID":"028ec67a-77a0-40aa-8d3b-1abc794287e7","Type":"ContainerStarted","Data":"12bb1769a1e47ac2e4ec49be649a94b7f6b0e5f5102a24430783525f5dd01a8f"} Nov 03 20:37:50.925413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.925307 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:37:50.988193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.988151 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" event={"ID":"e3aef707-29ed-4249-9d05-11c637aefdfb","Type":"ContainerDied","Data":"245adb8a4afbdcc39101d995e1aa865a5999aa05f1e8f9ae0c515e0059f4aa2f"} Nov 03 20:37:50.988487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.988466 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245adb8a4afbdcc39101d995e1aa865a5999aa05f1e8f9ae0c515e0059f4aa2f" Nov 03 20:37:50.988780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.988765 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6" Nov 03 20:37:51.003192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:50.993023 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" podStartSLOduration=1.993005112 podStartE2EDuration="1.993005112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:50.957984223 +0000 UTC m=+2180.488329899" watchObservedRunningTime="2026-11-03 20:37:50.993005112 +0000 UTC m=+2180.523350784" Nov 03 20:37:51.037215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:51.037181 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:51.405127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:51.405096 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk"] Nov 03 20:37:51.406881 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:51.406851 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaec364_a55c_4ca2_840c_05aa6f2a5a33.slice/crio-fd0f8bc047d5bcc262d30852c7c7166a598bba484befa95cbf155d6e468eea74 WatchSource:0}: Error finding container fd0f8bc047d5bcc262d30852c7c7166a598bba484befa95cbf155d6e468eea74: Status 404 returned error can't find the container with id fd0f8bc047d5bcc262d30852c7c7166a598bba484befa95cbf155d6e468eea74 Nov 03 20:37:52.008452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:52.008401 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" event={"ID":"caaec364-a55c-4ca2-840c-05aa6f2a5a33","Type":"ContainerStarted","Data":"0d6b825e55fc59190c71903d2b34129f3bacbdef3e8a76f530bf114e4bbb8559"} Nov 03 20:37:52.008893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:52.008478 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" event={"ID":"caaec364-a55c-4ca2-840c-05aa6f2a5a33","Type":"ContainerStarted","Data":"fd0f8bc047d5bcc262d30852c7c7166a598bba484befa95cbf155d6e468eea74"} Nov 03 20:37:52.030601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:52.030548 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" podStartSLOduration=2.030530462 podStartE2EDuration="2.030530462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:52.026928488 +0000 UTC m=+2181.557274161" watchObservedRunningTime="2026-11-03 20:37:52.030530462 +0000 UTC m=+2181.560876138" Nov 03 20:37:53.021816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:53.021778 5815 generic.go:356] "Generic (PLEG): container finished" podID="caaec364-a55c-4ca2-840c-05aa6f2a5a33" containerID="0d6b825e55fc59190c71903d2b34129f3bacbdef3e8a76f530bf114e4bbb8559" exitCode=0 Nov 03 20:37:53.022264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:53.021827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" event={"ID":"caaec364-a55c-4ca2-840c-05aa6f2a5a33","Type":"ContainerDied","Data":"0d6b825e55fc59190c71903d2b34129f3bacbdef3e8a76f530bf114e4bbb8559"} Nov 03 20:37:54.234731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.234703 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:54.267596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.267130 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn\" (UniqueName: \"kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn\") pod \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " Nov 03 20:37:54.267596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.267327 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca\") pod \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " Nov 03 20:37:54.267596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.267398 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca\") pod \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\" (UID: \"caaec364-a55c-4ca2-840c-05aa6f2a5a33\") " Nov 03 20:37:54.268241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.268208 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca" pod "caaec364-a55c-4ca2-840c-05aa6f2a5a33" (UID: "caaec364-a55c-4ca2-840c-05aa6f2a5a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:54.268457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.268433 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:54.270428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.270404 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn" pod "caaec364-a55c-4ca2-840c-05aa6f2a5a33" (UID: "caaec364-a55c-4ca2-840c-05aa6f2a5a33"). InnerVolumeSpecName "kube-api-access-2wwdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:54.270668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.270643 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca" pod "caaec364-a55c-4ca2-840c-05aa6f2a5a33" (UID: "caaec364-a55c-4ca2-840c-05aa6f2a5a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:54.369522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.369471 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwdn\" (UniqueName: \"kubernetes.io/projected/caaec364-a55c-4ca2-840c-05aa6f2a5a33-kube-api-access-2wwdn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:54.369667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:54.369542 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caaec364-a55c-4ca2-840c-05aa6f2a5a33-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:55.045008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:55.044970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" event={"ID":"caaec364-a55c-4ca2-840c-05aa6f2a5a33","Type":"ContainerDied","Data":"fd0f8bc047d5bcc262d30852c7c7166a598bba484befa95cbf155d6e468eea74"} Nov 03 20:37:55.045193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:55.045023 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk" Nov 03 20:37:55.045193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:55.045027 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0f8bc047d5bcc262d30852c7c7166a598bba484befa95cbf155d6e468eea74" Nov 03 20:37:57.067134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.067095 5815 generic.go:356] "Generic (PLEG): container finished" podID="e2041c14-e13f-4e68-8008-df150d920310" containerID="fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4" exitCode=0 Nov 03 20:37:57.067736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.067171 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" event={"ID":"e2041c14-e13f-4e68-8008-df150d920310","Type":"ContainerDied","Data":"fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4"} Nov 03 20:37:57.067834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.067734 5815 scope.go:122] "RemoveContainer" containerID="fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4" Nov 03 20:37:57.069727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.069690 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:37:57.069857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.069784 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:37:57.069857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.069836 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:37:57.745526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.745469 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw"] Nov 03 20:37:57.746774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.746752 5815 cpu_manager.go:414] "containerMap: removing container" podUID="caaec364-a55c-4ca2-840c-05aa6f2a5a33" containerName="revoke-keys" Nov 03 20:37:57.746774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.746774 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaec364-a55c-4ca2-840c-05aa6f2a5a33" containerName="revoke-keys" Nov 03 20:37:57.747068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.747054 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaec364-a55c-4ca2-840c-05aa6f2a5a33" containerName="revoke-keys" Nov 03 20:37:57.753760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.753736 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.755719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.755696 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw"] Nov 03 20:37:57.761525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.757969 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:37:57.766654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.766625 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:37:57.895702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.895658 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhf66\" (UniqueName: \"kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.895922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.895905 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.896028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.896008 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.997786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.997659 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.997786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.997760 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf66\" (UniqueName: \"kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.998055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.997900 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:57.999597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:57.999129 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:58.001180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.001153 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:58.010852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.010823 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf66\" (UniqueName: \"kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66\") pod \"maas-api-revoke-sub-3704215d-pzclw\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:58.104248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.104213 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" event={"ID":"e2041c14-e13f-4e68-8008-df150d920310","Type":"ContainerStarted","Data":"ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53"} Nov 03 20:37:58.127525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.127434 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:37:58.499979 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:37:58.499945 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39e5864_04b8_4ee6_ba19_f933c274b34e.slice/crio-9370fbd74346108c230c972b806cfc806243379ab52e0080f84d1cc8baa4a774 WatchSource:0}: Error finding container 9370fbd74346108c230c972b806cfc806243379ab52e0080f84d1cc8baa4a774: Status 404 returned error can't find the container with id 9370fbd74346108c230c972b806cfc806243379ab52e0080f84d1cc8baa4a774 Nov 03 20:37:58.556457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.556422 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw"] Nov 03 20:37:58.599690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.599648 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 20:37:58.599985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.599956 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 20:37:58.605805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:58.605782 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:37:59.118592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:59.118461 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" event={"ID":"a39e5864-04b8-4ee6-ba19-f933c274b34e","Type":"ContainerStarted","Data":"7d9904cefbe2ff836987486cbc7274bf95c568e222ec571cb5134172b8866f3d"} Nov 03 20:37:59.118592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:59.118550 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" event={"ID":"a39e5864-04b8-4ee6-ba19-f933c274b34e","Type":"ContainerStarted","Data":"9370fbd74346108c230c972b806cfc806243379ab52e0080f84d1cc8baa4a774"} Nov 03 20:37:59.220661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:59.219610 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" podStartSLOduration=2.219589025 podStartE2EDuration="2.219589025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:59.189565175 +0000 UTC m=+2188.719910852" watchObservedRunningTime="2026-11-03 20:37:59.219589025 +0000 UTC m=+2188.749934698" Nov 03 20:37:59.600136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:59.600104 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 20:37:59.600314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:59.600157 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 20:37:59.928589 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:37:59.926818 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39e5864_04b8_4ee6_ba19_f933c274b34e.slice/crio-7d9904cefbe2ff836987486cbc7274bf95c568e222ec571cb5134172b8866f3d.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:37:59.970576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:37:59.968721 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:38:00.272571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:00.262322 5815 generic.go:356] "Generic (PLEG): container finished" podID="a39e5864-04b8-4ee6-ba19-f933c274b34e" containerID="7d9904cefbe2ff836987486cbc7274bf95c568e222ec571cb5134172b8866f3d" exitCode=0 Nov 03 20:38:00.272571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:00.262391 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" event={"ID":"a39e5864-04b8-4ee6-ba19-f933c274b34e","Type":"ContainerDied","Data":"7d9904cefbe2ff836987486cbc7274bf95c568e222ec571cb5134172b8866f3d"} Nov 03 20:38:00.600616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:00.600531 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 20:38:00.600616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:00.600596 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 20:38:01.552357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.552327 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:38:01.601650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.601234 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 20:38:01.601650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.601296 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 20:38:01.701950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.701912 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66\" (UniqueName: \"kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66\") pod \"a39e5864-04b8-4ee6-ba19-f933c274b34e\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " Nov 03 20:38:01.702133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.701989 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca\") pod \"a39e5864-04b8-4ee6-ba19-f933c274b34e\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " Nov 03 20:38:01.702196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.702171 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca\") pod \"a39e5864-04b8-4ee6-ba19-f933c274b34e\" (UID: \"a39e5864-04b8-4ee6-ba19-f933c274b34e\") " Nov 03 20:38:01.702589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.702560 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca" pod "a39e5864-04b8-4ee6-ba19-f933c274b34e" (UID: "a39e5864-04b8-4ee6-ba19-f933c274b34e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:01.702758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.702737 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:01.704344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.704320 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca" pod "a39e5864-04b8-4ee6-ba19-f933c274b34e" (UID: "a39e5864-04b8-4ee6-ba19-f933c274b34e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:38:01.704593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.704572 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66" pod "a39e5864-04b8-4ee6-ba19-f933c274b34e" (UID: "a39e5864-04b8-4ee6-ba19-f933c274b34e"). InnerVolumeSpecName "kube-api-access-fhf66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:01.819996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.819952 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a39e5864-04b8-4ee6-ba19-f933c274b34e-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:01.819996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:01.820001 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhf66\" (UniqueName: \"kubernetes.io/projected/a39e5864-04b8-4ee6-ba19-f933c274b34e-kube-api-access-fhf66\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:02.042536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.040735 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:38:02.218391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.217346 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n"] Nov 03 20:38:02.317210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.317119 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" event={"ID":"a39e5864-04b8-4ee6-ba19-f933c274b34e","Type":"ContainerDied","Data":"9370fbd74346108c230c972b806cfc806243379ab52e0080f84d1cc8baa4a774"} Nov 03 20:38:02.317210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.317183 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9370fbd74346108c230c972b806cfc806243379ab52e0080f84d1cc8baa4a774" Nov 03 20:38:02.317458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.317257 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw" Nov 03 20:38:02.613240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.609949 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 20:38:02.613240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:02.610048 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 20:38:03.394392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:03.394359 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:38:03.600335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:03.600296 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 20:38:03.600552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:03.600370 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 20:38:04.604524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:04.604470 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:38:04.605770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:04.605749 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 20:38:07.218669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.218632 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" podUID="af62faa8-b63c-4e66-85fe-354238e0b6ef" containerName="payload-processing" containerID="cri-o://be0129b04d03a40a425a2b3e49a37a932a4b6245bc540acf22e6226a4936a7f1" gracePeriod=25 Nov 03 20:38:07.439763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.439721 5815 generic.go:356] "Generic (PLEG): container finished" podID="af62faa8-b63c-4e66-85fe-354238e0b6ef" containerID="be0129b04d03a40a425a2b3e49a37a932a4b6245bc540acf22e6226a4936a7f1" exitCode=0 Nov 03 20:38:07.440465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.440431 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" event={"ID":"af62faa8-b63c-4e66-85fe-354238e0b6ef","Type":"ContainerDied","Data":"be0129b04d03a40a425a2b3e49a37a932a4b6245bc540acf22e6226a4936a7f1"} Nov 03 20:38:07.596162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.596130 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:38:07.640268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.640237 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume\") pod \"af62faa8-b63c-4e66-85fe-354238e0b6ef\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " Nov 03 20:38:07.640462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.640335 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5\") pod \"af62faa8-b63c-4e66-85fe-354238e0b6ef\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " Nov 03 20:38:07.640462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.640383 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume\") pod \"af62faa8-b63c-4e66-85fe-354238e0b6ef\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " Nov 03 20:38:07.640648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.640483 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials\") pod \"af62faa8-b63c-4e66-85fe-354238e0b6ef\" (UID: \"af62faa8-b63c-4e66-85fe-354238e0b6ef\") " Nov 03 20:38:07.640804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.640774 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume" pod "af62faa8-b63c-4e66-85fe-354238e0b6ef" (UID: "af62faa8-b63c-4e66-85fe-354238e0b6ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:07.640980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.640929 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume" pod "af62faa8-b63c-4e66-85fe-354238e0b6ef" (UID: "af62faa8-b63c-4e66-85fe-354238e0b6ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:07.642589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.642569 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:07.646916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.646892 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af62faa8-b63c-4e66-85fe-354238e0b6ef-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:07.646916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.643685 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials" pod "af62faa8-b63c-4e66-85fe-354238e0b6ef" (UID: "af62faa8-b63c-4e66-85fe-354238e0b6ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:07.647095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.643745 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5" pod "af62faa8-b63c-4e66-85fe-354238e0b6ef" (UID: "af62faa8-b63c-4e66-85fe-354238e0b6ef"). InnerVolumeSpecName "kube-api-access-f4mg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:07.748960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.748917 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mg5\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-kube-api-access-f4mg5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:07.749236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:07.749214 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af62faa8-b63c-4e66-85fe-354238e0b6ef-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:08.452142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.452109 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" event={"ID":"af62faa8-b63c-4e66-85fe-354238e0b6ef","Type":"ContainerDied","Data":"0b4b4733c257eaf5a1e6edcd521267f399d1322774ad477d98328a69c3c31324"} Nov 03 20:38:08.452633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.452178 5815 scope.go:122] "RemoveContainer" containerID="be0129b04d03a40a425a2b3e49a37a932a4b6245bc540acf22e6226a4936a7f1" Nov 03 20:38:08.452633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.452185 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n" Nov 03 20:38:08.481865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.481841 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n"] Nov 03 20:38:08.487021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.486996 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-748fpc62n"] Nov 03 20:38:08.592676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.592640 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af62faa8-b63c-4e66-85fe-354238e0b6ef" path="/var/lib/kubelet/pods/af62faa8-b63c-4e66-85fe-354238e0b6ef/volumes" Nov 03 20:38:08.719191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:08.719157 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq"] Nov 03 20:38:12.388641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.388571 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6"] Nov 03 20:38:12.390388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.390360 5815 cpu_manager.go:414] "containerMap: removing container" podUID="af62faa8-b63c-4e66-85fe-354238e0b6ef" containerName="payload-processing" Nov 03 20:38:12.390601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.390582 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af62faa8-b63c-4e66-85fe-354238e0b6ef" containerName="payload-processing" Nov 03 20:38:12.390734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.390721 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a39e5864-04b8-4ee6-ba19-f933c274b34e" containerName="revoke-keys" Nov 03 20:38:12.390823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.390811 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39e5864-04b8-4ee6-ba19-f933c274b34e" containerName="revoke-keys" Nov 03 20:38:12.406531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.391161 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="af62faa8-b63c-4e66-85fe-354238e0b6ef" containerName="payload-processing" Nov 03 20:38:12.406764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.406745 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39e5864-04b8-4ee6-ba19-f933c274b34e" containerName="revoke-keys" Nov 03 20:38:12.417537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.413048 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.449538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.437288 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6"] Nov 03 20:38:12.720778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.632846 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.721207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.721173 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6nf\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.722725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.721546 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.723079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.722999 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.825209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.825172 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6nf\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.825388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.825275 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.825388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.825355 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.825544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.825496 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.825998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.825966 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.826117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.826097 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.828044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.828022 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.837684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.837662 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6nf\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:12.858664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:12.858641 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:13.379861 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:38:13.379834 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3109641a_4af2_4e4a_92a8_9745c396b189.slice/crio-870a41810ecfc1e8828331e0f13ac8933f35f6d578d5010884e5d4e0b906c514 WatchSource:0}: Error finding container 870a41810ecfc1e8828331e0f13ac8933f35f6d578d5010884e5d4e0b906c514: Status 404 returned error can't find the container with id 870a41810ecfc1e8828331e0f13ac8933f35f6d578d5010884e5d4e0b906c514 Nov 03 20:38:13.380955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:13.380892 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6"] Nov 03 20:38:13.720406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:13.720366 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" podUID="959f9758-18fa-4a15-87fe-eb450198a4d4" containerName="payload-processing" containerID="cri-o://f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290" gracePeriod=25 Nov 03 20:38:13.748881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:13.748850 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" event={"ID":"3109641a-4af2-4e4a-92a8-9745c396b189","Type":"ContainerStarted","Data":"9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3"} Nov 03 20:38:13.749031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:13.748912 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" event={"ID":"3109641a-4af2-4e4a-92a8-9745c396b189","Type":"ContainerStarted","Data":"870a41810ecfc1e8828331e0f13ac8933f35f6d578d5010884e5d4e0b906c514"} Nov 03 20:38:13.749031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:13.748942 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:13.776482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:13.776426 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" podStartSLOduration=1.7764081040000002 podStartE2EDuration="1.776408104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:13.769638504 +0000 UTC m=+2203.299984177" watchObservedRunningTime="2026-11-03 20:38:13.776408104 +0000 UTC m=+2203.306753778" Nov 03 20:38:13.855497 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:38:13.855465 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:38:14.102996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.102969 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:38:14.170189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.170155 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials\") pod \"959f9758-18fa-4a15-87fe-eb450198a4d4\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " Nov 03 20:38:14.170367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.170349 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume\") pod \"959f9758-18fa-4a15-87fe-eb450198a4d4\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " Nov 03 20:38:14.170492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.170477 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt\") pod \"959f9758-18fa-4a15-87fe-eb450198a4d4\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " Nov 03 20:38:14.170573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.170554 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume\") pod \"959f9758-18fa-4a15-87fe-eb450198a4d4\" (UID: \"959f9758-18fa-4a15-87fe-eb450198a4d4\") " Nov 03 20:38:14.170808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.170777 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume" pod "959f9758-18fa-4a15-87fe-eb450198a4d4" (UID: "959f9758-18fa-4a15-87fe-eb450198a4d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:14.171256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.171159 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume" pod "959f9758-18fa-4a15-87fe-eb450198a4d4" (UID: "959f9758-18fa-4a15-87fe-eb450198a4d4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:14.171256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.171273 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:14.172785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.172761 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials" pod "959f9758-18fa-4a15-87fe-eb450198a4d4" (UID: "959f9758-18fa-4a15-87fe-eb450198a4d4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:14.172957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.172865 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt" pod "959f9758-18fa-4a15-87fe-eb450198a4d4" (UID: "959f9758-18fa-4a15-87fe-eb450198a4d4"). InnerVolumeSpecName "kube-api-access-xwsbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:14.272541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.272496 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwsbt\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-kube-api-access-xwsbt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:14.272541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.272539 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/959f9758-18fa-4a15-87fe-eb450198a4d4-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:14.272787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.272550 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/959f9758-18fa-4a15-87fe-eb450198a4d4-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:14.759324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.759290 5815 generic.go:356] "Generic (PLEG): container finished" podID="959f9758-18fa-4a15-87fe-eb450198a4d4" containerID="f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290" exitCode=0 Nov 03 20:38:14.759802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.759336 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" Nov 03 20:38:14.759802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.759371 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" event={"ID":"959f9758-18fa-4a15-87fe-eb450198a4d4","Type":"ContainerDied","Data":"f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290"} Nov 03 20:38:14.759802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.759426 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq" event={"ID":"959f9758-18fa-4a15-87fe-eb450198a4d4","Type":"ContainerDied","Data":"f83c1458583589b95649a176d333847d5b667c4557967295104a261e375f8dc9"} Nov 03 20:38:14.759802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.759453 5815 scope.go:122] "RemoveContainer" containerID="f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290" Nov 03 20:38:14.770947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.770928 5815 scope.go:122] "RemoveContainer" containerID="f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290" Nov 03 20:38:14.771242 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:38:14.771223 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290\": container with ID starting with f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290 not found: ID does not exist" containerID="f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290" Nov 03 20:38:14.771322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.771250 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290"} err="failed to get container status \"f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290\": rpc error: code = NotFound desc = could not find container \"f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290\": container with ID starting with f1a3b2cb32df3b46eb625436521f9f695bcc3f8cdfa8d4e08daf39c82cde9290 not found: ID does not exist" Nov 03 20:38:14.783573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.783553 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq"] Nov 03 20:38:14.788991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:14.788973 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65lzsfq"] Nov 03 20:38:16.587795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:16.587763 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959f9758-18fa-4a15-87fe-eb450198a4d4" path="/var/lib/kubelet/pods/959f9758-18fa-4a15-87fe-eb450198a4d4/volumes" Nov 03 20:38:18.788002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.787957 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p"] Nov 03 20:38:18.789231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.789203 5815 cpu_manager.go:414] "containerMap: removing container" podUID="959f9758-18fa-4a15-87fe-eb450198a4d4" containerName="payload-processing" Nov 03 20:38:18.789396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.789232 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959f9758-18fa-4a15-87fe-eb450198a4d4" containerName="payload-processing" Nov 03 20:38:18.789450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.789420 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="959f9758-18fa-4a15-87fe-eb450198a4d4" containerName="payload-processing" Nov 03 20:38:18.795071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.795051 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:18.804139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.804117 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"kube-root-ca.crt\"" Nov 03 20:38:18.804609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.804405 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-14ebbb\"" Nov 03 20:38:18.804763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.804440 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"openshift-service-ca.crt\"" Nov 03 20:38:18.804898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.804494 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"routing-overlay\"" Nov 03 20:38:18.805157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.805140 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p"] Nov 03 20:38:18.881480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.881447 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"agc-cred-6e08feeda55f\"" Nov 03 20:38:18.881799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.881777 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"payload-processing-external-model-e2e--6b5dcb9f-dockercfg-hbgzq\"" Nov 03 20:38:18.918609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.918566 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:18.918792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.918622 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5227r\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:18.918870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.918844 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:18.918967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:18.918948 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.021165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.021063 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.021165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.021154 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.021491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.021473 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.021638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.021529 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5227r\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.022120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.022090 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.022262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.022058 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.024302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.024270 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.034055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.034027 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5227r\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.135828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.135790 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:19.495531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.492862 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d"] Nov 03 20:38:19.521531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.518286 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d"] Nov 03 20:38:19.521531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.518419 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.529532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.526448 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"agc-cred-09a9bbf37990\"" Nov 03 20:38:19.614009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.611921 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.614009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.612212 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vqr\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.614009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.612288 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.614009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.612361 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.714531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.714455 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vqr\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.714760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.714545 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.714760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.714599 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.714760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.714667 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.723533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.716396 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.723533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.716409 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.726254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.726222 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.729137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.729115 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vqr\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:19.827789 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:38:19.827761 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be22f3b_6062_41ee_9f44_5ef00a5becc0.slice/crio-557e43d4355241a75ae53c033e010bd1df0d70feb6c52a32e8545151cb7fdc58 WatchSource:0}: Error finding container 557e43d4355241a75ae53c033e010bd1df0d70feb6c52a32e8545151cb7fdc58: Status 404 returned error can't find the container with id 557e43d4355241a75ae53c033e010bd1df0d70feb6c52a32e8545151cb7fdc58 Nov 03 20:38:19.828657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.828619 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p"] Nov 03 20:38:19.892531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:19.890890 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:20.426641 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:38:20.426612 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f224d64_55fc_4fd9_acf9_55a1e08569b4.slice/crio-9f5e152cd7f9ea7663a27ecc1e9d2841e759920597d32949f3cbf3c9a1eb8460 WatchSource:0}: Error finding container 9f5e152cd7f9ea7663a27ecc1e9d2841e759920597d32949f3cbf3c9a1eb8460: Status 404 returned error can't find the container with id 9f5e152cd7f9ea7663a27ecc1e9d2841e759920597d32949f3cbf3c9a1eb8460 Nov 03 20:38:20.428006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.427980 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d"] Nov 03 20:38:20.830166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.830132 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" event={"ID":"5be22f3b-6062-41ee-9f44-5ef00a5becc0","Type":"ContainerStarted","Data":"be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51"} Nov 03 20:38:20.830652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.830196 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:20.830652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.830221 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" event={"ID":"5be22f3b-6062-41ee-9f44-5ef00a5becc0","Type":"ContainerStarted","Data":"557e43d4355241a75ae53c033e010bd1df0d70feb6c52a32e8545151cb7fdc58"} Nov 03 20:38:20.832247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.832224 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" event={"ID":"8f224d64-55fc-4fd9-acf9-55a1e08569b4","Type":"ContainerStarted","Data":"a6ef4f03da31bb877add8e02e3764a30fca9c2d662e571b2f35bc5fb5430d742"} Nov 03 20:38:20.832373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.832262 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" event={"ID":"8f224d64-55fc-4fd9-acf9-55a1e08569b4","Type":"ContainerStarted","Data":"9f5e152cd7f9ea7663a27ecc1e9d2841e759920597d32949f3cbf3c9a1eb8460"} Nov 03 20:38:20.832373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.832282 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:20.851555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:20.851493 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" podStartSLOduration=2.8514794610000003 podStartE2EDuration="2.851479461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:20.849262409 +0000 UTC m=+2210.379608081" watchObservedRunningTime="2026-11-03 20:38:20.851479461 +0000 UTC m=+2210.381825133" Nov 03 20:38:22.445906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.408331 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" podStartSLOduration=3.408299577 podStartE2EDuration="3.408299577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:20.90090892 +0000 UTC m=+2210.431254595" watchObservedRunningTime="2026-11-03 20:38:22.408299577 +0000 UTC m=+2211.938645252" Nov 03 20:38:22.458960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.458930 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6"] Nov 03 20:38:22.462883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.462861 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:22.468534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.467554 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq"] Nov 03 20:38:22.477945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.477923 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.481288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.481263 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq"] Nov 03 20:38:22.507529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.506940 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-07c1d8207764\"" Nov 03 20:38:22.557381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.557350 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.557591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.557570 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.557696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.557677 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.557819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.557806 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qt2\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.659090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.659055 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qt2\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.659264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.659137 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.659264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.659247 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.659347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.659312 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.659932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.659910 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.660017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.659999 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.662280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.662256 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.701574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.699415 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qt2\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:22.824200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:22.824165 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:23.243008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:23.242976 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq"] Nov 03 20:38:23.246052 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:38:23.246009 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4618239_7269_4c26_83db_bc16f831eada.slice/crio-4ce3983eaa92e8ae32c6a00f10013296060722184307e0bf269c5b82a1bdaadf WatchSource:0}: Error finding container 4ce3983eaa92e8ae32c6a00f10013296060722184307e0bf269c5b82a1bdaadf: Status 404 returned error can't find the container with id 4ce3983eaa92e8ae32c6a00f10013296060722184307e0bf269c5b82a1bdaadf Nov 03 20:38:23.944473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:23.944434 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" event={"ID":"b4618239-7269-4c26-83db-bc16f831eada","Type":"ContainerStarted","Data":"38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547"} Nov 03 20:38:23.950150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:23.950129 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:23.950377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:23.950361 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" event={"ID":"b4618239-7269-4c26-83db-bc16f831eada","Type":"ContainerStarted","Data":"4ce3983eaa92e8ae32c6a00f10013296060722184307e0bf269c5b82a1bdaadf"} Nov 03 20:38:24.205213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.205093 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" podStartSLOduration=2.205070956 podStartE2EDuration="2.205070956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:24.176889718 +0000 UTC m=+2213.707235390" watchObservedRunningTime="2026-11-03 20:38:24.205070956 +0000 UTC m=+2213.735416629" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.481558 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f"] Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.488119 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f"] Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.488233 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.496950 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"kube-root-ca.crt\"" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.497184 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"payload-processing-external-model-e2e--4cfe9bda-dockercfg-c9p8h\"" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.497255 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"e2e-cmp-legacy-a2f02d-provider-cred\"" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.497333 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"openshift-service-ca.crt\"" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.497391 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dce009\"" Nov 03 20:38:24.509338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.501989 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"routing-overlay\"" Nov 03 20:38:24.648126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.636629 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.648126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.636887 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.648126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.636955 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.648126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.636999 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nd84\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.816908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.811583 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.816908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.815682 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.816908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.815777 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.816908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.815820 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nd84\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.836533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.828994 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.853537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.845542 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.916591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.910526 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.948335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.940271 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nd84\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:24.953261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:24.952918 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:38:25.129697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:25.129622 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:25.450725 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:38:25.450693 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d6e20e_77cc_41bc_9b10_7b051c0111db.slice/crio-72ff93c4edde2b1f61a01e268689c4d6ab65c61aa128df397ed0daad6f322966 WatchSource:0}: Error finding container 72ff93c4edde2b1f61a01e268689c4d6ab65c61aa128df397ed0daad6f322966: Status 404 returned error can't find the container with id 72ff93c4edde2b1f61a01e268689c4d6ab65c61aa128df397ed0daad6f322966 Nov 03 20:38:25.451165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:25.451142 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f"] Nov 03 20:38:26.004151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:26.004116 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" event={"ID":"26d6e20e-77cc-41bc-9b10-7b051c0111db","Type":"ContainerStarted","Data":"5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93"} Nov 03 20:38:26.004619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:26.004172 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" event={"ID":"26d6e20e-77cc-41bc-9b10-7b051c0111db","Type":"ContainerStarted","Data":"72ff93c4edde2b1f61a01e268689c4d6ab65c61aa128df397ed0daad6f322966"} Nov 03 20:38:26.004619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:26.004195 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:26.055040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:26.054976 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" podStartSLOduration=2.054957559 podStartE2EDuration="2.054957559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:26.053315939 +0000 UTC m=+2215.583661610" watchObservedRunningTime="2026-11-03 20:38:26.054957559 +0000 UTC m=+2215.585303233" Nov 03 20:38:27.460687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.460625 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" podUID="3109641a-4af2-4e4a-92a8-9745c396b189" containerName="payload-processing" containerID="cri-o://9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3" gracePeriod=25 Nov 03 20:38:27.767734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.767706 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:27.881271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.881232 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf\") pod \"3109641a-4af2-4e4a-92a8-9745c396b189\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " Nov 03 20:38:27.881483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.881313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials\") pod \"3109641a-4af2-4e4a-92a8-9745c396b189\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " Nov 03 20:38:27.881483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.881361 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume\") pod \"3109641a-4af2-4e4a-92a8-9745c396b189\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " Nov 03 20:38:27.881724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.881700 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume\") pod \"3109641a-4af2-4e4a-92a8-9745c396b189\" (UID: \"3109641a-4af2-4e4a-92a8-9745c396b189\") " Nov 03 20:38:27.881856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.881829 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume" pod "3109641a-4af2-4e4a-92a8-9745c396b189" (UID: "3109641a-4af2-4e4a-92a8-9745c396b189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:27.882029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.882012 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume" pod "3109641a-4af2-4e4a-92a8-9745c396b189" (UID: "3109641a-4af2-4e4a-92a8-9745c396b189"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:27.882381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.882363 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:27.882439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.882384 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3109641a-4af2-4e4a-92a8-9745c396b189-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:27.884360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.884337 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf" pod "3109641a-4af2-4e4a-92a8-9745c396b189" (UID: "3109641a-4af2-4e4a-92a8-9745c396b189"). InnerVolumeSpecName "kube-api-access-gg6nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:27.884442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.884370 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials" pod "3109641a-4af2-4e4a-92a8-9745c396b189" (UID: "3109641a-4af2-4e4a-92a8-9745c396b189"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:27.984378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.984289 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg6nf\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-kube-api-access-gg6nf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:27.984378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:27.984318 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3109641a-4af2-4e4a-92a8-9745c396b189-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:28.062412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.062380 5815 generic.go:356] "Generic (PLEG): container finished" podID="3109641a-4af2-4e4a-92a8-9745c396b189" containerID="9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3" exitCode=0 Nov 03 20:38:28.062646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.062444 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" Nov 03 20:38:28.062646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.062474 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" event={"ID":"3109641a-4af2-4e4a-92a8-9745c396b189","Type":"ContainerDied","Data":"9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3"} Nov 03 20:38:28.062646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.062555 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6" event={"ID":"3109641a-4af2-4e4a-92a8-9745c396b189","Type":"ContainerDied","Data":"870a41810ecfc1e8828331e0f13ac8933f35f6d578d5010884e5d4e0b906c514"} Nov 03 20:38:28.062646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.062579 5815 scope.go:122] "RemoveContainer" containerID="9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3" Nov 03 20:38:28.079161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.079140 5815 scope.go:122] "RemoveContainer" containerID="9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3" Nov 03 20:38:28.079639 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:38:28.079614 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3\": container with ID starting with 9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3 not found: ID does not exist" containerID="9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3" Nov 03 20:38:28.079741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.079649 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3"} err="failed to get container status \"9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3\": rpc error: code = NotFound desc = could not find container \"9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3\": container with ID starting with 9f9cbbe1d88c4ba9a86262dcd48d34b460d09f719a6e934992959578c3180ba3 not found: ID does not exist" Nov 03 20:38:28.099838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.099795 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6"] Nov 03 20:38:28.104533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.104491 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc99fcj6"] Nov 03 20:38:28.592032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:28.591996 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3109641a-4af2-4e4a-92a8-9745c396b189" path="/var/lib/kubelet/pods/3109641a-4af2-4e4a-92a8-9745c396b189/volumes" Nov 03 20:38:31.877928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:31.877898 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:31.878496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:31.877981 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:38:31.955479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:31.955442 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p"] Nov 03 20:38:32.149566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:32.149467 5815 generic.go:356] "Generic (PLEG): container finished" podID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerID="00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264" exitCode=0 Nov 03 20:38:32.149566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:32.149548 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" event={"ID":"ac6976b5-e1d9-442b-9b8f-a06b6edf290b","Type":"ContainerDied","Data":"00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264"} Nov 03 20:38:32.150082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:32.150066 5815 scope.go:122] "RemoveContainer" containerID="00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264" Nov 03 20:38:32.151951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:32.151923 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:38:32.152013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:32.151994 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:38:32.152050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:32.152026 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:38:33.160707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:33.160669 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" event={"ID":"ac6976b5-e1d9-442b-9b8f-a06b6edf290b","Type":"ContainerStarted","Data":"d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27"} Nov 03 20:38:34.010044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:34.010002 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:38:34.011133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:34.011099 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 03 20:38:34.011282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:34.011167 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 03 20:38:35.010172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:35.010126 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 03 20:38:35.010734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:35.010181 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 03 20:38:36.008160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:36.008128 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:38:36.009984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:36.009959 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 03 20:38:36.010091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:36.010016 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 03 20:38:36.090535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:36.089167 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw"] Nov 03 20:38:37.010457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.010409 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 03 20:38:37.010698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.010497 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 03 20:38:37.053566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.053540 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:38:37.150025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.149986 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" podUID="5be22f3b-6062-41ee-9f44-5ef00a5becc0" containerName="payload-processing" containerID="cri-o://be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51" gracePeriod=25 Nov 03 20:38:37.258530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.254590 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:38:37.340359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.337212 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49"] Nov 03 20:38:37.340359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.338370 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3109641a-4af2-4e4a-92a8-9745c396b189" containerName="payload-processing" Nov 03 20:38:37.340359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.338391 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3109641a-4af2-4e4a-92a8-9745c396b189" containerName="payload-processing" Nov 03 20:38:37.340359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.338646 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3109641a-4af2-4e4a-92a8-9745c396b189" containerName="payload-processing" Nov 03 20:38:37.346324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.344340 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.346324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.346246 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49"] Nov 03 20:38:37.354588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.354548 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-03c633-openshift-default-dockercfg-4k2z2\"" Nov 03 20:38:37.461586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.461491 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.461761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.461699 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.461838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.461770 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.461838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.461810 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.461957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.461887 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.461957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.461929 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.462074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.462049 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.462134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.462085 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.462134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.462113 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7vx\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563273 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563321 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563349 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7vx\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563440 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563613 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563679 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563720 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563794 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.563942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.563834 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.570542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.564423 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.600393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.575781 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.600393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.576063 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.600393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.586842 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.601153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.601102 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.645196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.645167 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.645825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.645565 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.646227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.646204 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.648276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.648234 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7vx\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.703394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.688845 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:37.735040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.735009 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:37.934222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.934127 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r\") pod \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " Nov 03 20:38:37.934418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.934278 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume\") pod \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " Nov 03 20:38:37.934418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.934332 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume\") pod \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " Nov 03 20:38:37.934560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.934420 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials\") pod \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\" (UID: \"5be22f3b-6062-41ee-9f44-5ef00a5becc0\") " Nov 03 20:38:37.936018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.935107 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume" pod "5be22f3b-6062-41ee-9f44-5ef00a5becc0" (UID: "5be22f3b-6062-41ee-9f44-5ef00a5becc0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:37.936018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.935225 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:37.936018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.935901 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume" pod "5be22f3b-6062-41ee-9f44-5ef00a5becc0" (UID: "5be22f3b-6062-41ee-9f44-5ef00a5becc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:37.938050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.938024 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r" pod "5be22f3b-6062-41ee-9f44-5ef00a5becc0" (UID: "5be22f3b-6062-41ee-9f44-5ef00a5becc0"). InnerVolumeSpecName "kube-api-access-5227r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:37.939331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:37.939291 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials" pod "5be22f3b-6062-41ee-9f44-5ef00a5becc0" (UID: "5be22f3b-6062-41ee-9f44-5ef00a5becc0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:38.010208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.010045 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Nov 03 20:38:38.010208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.010114 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Nov 03 20:38:38.036360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.036334 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be22f3b-6062-41ee-9f44-5ef00a5becc0-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:38.036360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.036359 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:38.036602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.036370 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5227r\" (UniqueName: \"kubernetes.io/projected/5be22f3b-6062-41ee-9f44-5ef00a5becc0-kube-api-access-5227r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:38.144535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.140280 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49"] Nov 03 20:38:38.156066 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:38:38.156015 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275de0e5_b391_42e6_9a41_641ae3e52d3a.slice/crio-e6a002f3c52c87af4ddbdb54cef93577b461c99558cae32cff7c107e9b506c00 WatchSource:0}: Error finding container e6a002f3c52c87af4ddbdb54cef93577b461c99558cae32cff7c107e9b506c00: Status 404 returned error can't find the container with id e6a002f3c52c87af4ddbdb54cef93577b461c99558cae32cff7c107e9b506c00 Nov 03 20:38:38.173177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.163263 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:38:38.173177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.163353 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:38:38.173177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.163399 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:38:38.345890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.345856 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" event={"ID":"275de0e5-b391-42e6-9a41-641ae3e52d3a","Type":"ContainerStarted","Data":"5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5"} Nov 03 20:38:38.346031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.345919 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" event={"ID":"275de0e5-b391-42e6-9a41-641ae3e52d3a","Type":"ContainerStarted","Data":"e6a002f3c52c87af4ddbdb54cef93577b461c99558cae32cff7c107e9b506c00"} Nov 03 20:38:38.347364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.347338 5815 generic.go:356] "Generic (PLEG): container finished" podID="5be22f3b-6062-41ee-9f44-5ef00a5becc0" containerID="be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51" exitCode=0 Nov 03 20:38:38.347486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.347395 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" event={"ID":"5be22f3b-6062-41ee-9f44-5ef00a5becc0","Type":"ContainerDied","Data":"be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51"} Nov 03 20:38:38.347486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.347442 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" event={"ID":"5be22f3b-6062-41ee-9f44-5ef00a5becc0","Type":"ContainerDied","Data":"557e43d4355241a75ae53c033e010bd1df0d70feb6c52a32e8545151cb7fdc58"} Nov 03 20:38:38.347486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.347464 5815 scope.go:122] "RemoveContainer" containerID="be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51" Nov 03 20:38:38.347650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.347614 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p" Nov 03 20:38:38.397528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.379686 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podStartSLOduration=1.379665769 podStartE2EDuration="1.379665769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:38.370880063 +0000 UTC m=+2227.901225743" watchObservedRunningTime="2026-11-03 20:38:38.379665769 +0000 UTC m=+2227.910011446" Nov 03 20:38:38.412621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.412058 5815 scope.go:122] "RemoveContainer" containerID="be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51" Nov 03 20:38:38.414032 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:38:38.414000 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51\": container with ID starting with be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51 not found: ID does not exist" containerID="be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51" Nov 03 20:38:38.414249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.414218 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51"} err="failed to get container status \"be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51\": rpc error: code = NotFound desc = could not find container \"be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51\": container with ID starting with be6d270c02f016db9cb027f7acaa9f1b546eec8a4f8b0a7b705f93b0b20dfe51 not found: ID does not exist" Nov 03 20:38:38.420234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.420212 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p"] Nov 03 20:38:38.424341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.424320 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf8ss86p"] Nov 03 20:38:38.587785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.587752 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be22f3b-6062-41ee-9f44-5ef00a5becc0" path="/var/lib/kubelet/pods/5be22f3b-6062-41ee-9f44-5ef00a5becc0/volumes" Nov 03 20:38:38.690677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:38.690645 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:39.024733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:39.024613 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:38:39.028520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:39.028479 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 20:38:39.694154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:39.694107 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:38:39.694778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:39.694179 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:38:40.694533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:40.693917 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:38:40.694533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:40.693997 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:38:41.091627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.091578 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" podUID="028ec67a-77a0-40aa-8d3b-1abc794287e7" containerName="payload-processing" containerID="cri-o://8f998d13e5cb55185178e6d6393190683418e6bdb0a0400c8ec554c2c8a5ea8e" gracePeriod=25 Nov 03 20:38:41.429530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.429194 5815 generic.go:356] "Generic (PLEG): container finished" podID="028ec67a-77a0-40aa-8d3b-1abc794287e7" containerID="8f998d13e5cb55185178e6d6393190683418e6bdb0a0400c8ec554c2c8a5ea8e" exitCode=0 Nov 03 20:38:41.429530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.429258 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" event={"ID":"028ec67a-77a0-40aa-8d3b-1abc794287e7","Type":"ContainerDied","Data":"8f998d13e5cb55185178e6d6393190683418e6bdb0a0400c8ec554c2c8a5ea8e"} Nov 03 20:38:41.523292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.523250 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:38:41.621267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.621178 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials\") pod \"028ec67a-77a0-40aa-8d3b-1abc794287e7\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " Nov 03 20:38:41.621444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.621322 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume\") pod \"028ec67a-77a0-40aa-8d3b-1abc794287e7\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " Nov 03 20:38:41.621500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.621483 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume\") pod \"028ec67a-77a0-40aa-8d3b-1abc794287e7\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " Nov 03 20:38:41.621646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.621627 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr\") pod \"028ec67a-77a0-40aa-8d3b-1abc794287e7\" (UID: \"028ec67a-77a0-40aa-8d3b-1abc794287e7\") " Nov 03 20:38:41.622202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.622104 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume" pod "028ec67a-77a0-40aa-8d3b-1abc794287e7" (UID: "028ec67a-77a0-40aa-8d3b-1abc794287e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:41.622525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.622480 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume" pod "028ec67a-77a0-40aa-8d3b-1abc794287e7" (UID: "028ec67a-77a0-40aa-8d3b-1abc794287e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:38:41.622640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.622623 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:41.624034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.624004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials" pod "028ec67a-77a0-40aa-8d3b-1abc794287e7" (UID: "028ec67a-77a0-40aa-8d3b-1abc794287e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:41.624835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.624816 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr" pod "028ec67a-77a0-40aa-8d3b-1abc794287e7" (UID: "028ec67a-77a0-40aa-8d3b-1abc794287e7"). InnerVolumeSpecName "kube-api-access-gr9zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:41.698224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.698185 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:38:41.698930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.698899 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:38:41.723881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.723852 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9zr\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-kube-api-access-gr9zr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:41.723881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.723878 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/028ec67a-77a0-40aa-8d3b-1abc794287e7-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:41.724051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:41.723889 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/028ec67a-77a0-40aa-8d3b-1abc794287e7-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:42.440005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.439962 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" event={"ID":"028ec67a-77a0-40aa-8d3b-1abc794287e7","Type":"ContainerDied","Data":"12bb1769a1e47ac2e4ec49be649a94b7f6b0e5f5102a24430783525f5dd01a8f"} Nov 03 20:38:42.440005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.439979 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw" Nov 03 20:38:42.440278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.440032 5815 scope.go:122] "RemoveContainer" containerID="8f998d13e5cb55185178e6d6393190683418e6bdb0a0400c8ec554c2c8a5ea8e" Nov 03 20:38:42.487586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.487556 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw"] Nov 03 20:38:42.500787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.500751 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcxtwhw"] Nov 03 20:38:42.604999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.604957 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028ec67a-77a0-40aa-8d3b-1abc794287e7" path="/var/lib/kubelet/pods/028ec67a-77a0-40aa-8d3b-1abc794287e7/volumes" Nov 03 20:38:42.692991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.692899 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:38:42.693271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:42.693242 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:38:43.701186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:43.701151 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:38:43.701827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:43.701794 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:38:43.768578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:43.768545 5815 scope.go:122] "RemoveContainer" containerID="fa9d56580ba37dcf83e41a7d8b3fd78169dd01b0b58690ac4fb2c5e458f83073" Nov 03 20:38:43.857003 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:38:43.856961 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:38:44.690283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:44.690245 5815 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body= Nov 03 20:38:44.690474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:44.690304 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" Nov 03 20:38:45.696978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:45.696938 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:45.697476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:45.697397 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:38:45.698114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:38:45.698096 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:39:07.704077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.704045 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69"] Nov 03 20:39:07.704917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.704898 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5be22f3b-6062-41ee-9f44-5ef00a5becc0" containerName="payload-processing" Nov 03 20:39:07.704917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.704919 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be22f3b-6062-41ee-9f44-5ef00a5becc0" containerName="payload-processing" Nov 03 20:39:07.705047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.704950 5815 cpu_manager.go:414] "containerMap: removing container" podUID="028ec67a-77a0-40aa-8d3b-1abc794287e7" containerName="payload-processing" Nov 03 20:39:07.705047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.704957 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028ec67a-77a0-40aa-8d3b-1abc794287e7" containerName="payload-processing" Nov 03 20:39:07.705119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.705073 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be22f3b-6062-41ee-9f44-5ef00a5becc0" containerName="payload-processing" Nov 03 20:39:07.705119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.705086 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="028ec67a-77a0-40aa-8d3b-1abc794287e7" containerName="payload-processing" Nov 03 20:39:07.710333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.710313 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.716229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.715139 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:07.716229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.715294 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:39:07.718465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.718442 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69"] Nov 03 20:39:07.780335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.780301 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xgn\" (UniqueName: \"kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.780521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.780448 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.780521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.780475 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.882622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.882587 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xgn\" (UniqueName: \"kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.882806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.882692 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.882806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.882711 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.883430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.883394 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.885393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.885372 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:07.895448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:07.895423 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xgn\" (UniqueName: \"kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn\") pod \"maas-api-revoke-sub-776f3f71-jws69\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:08.032733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:08.032698 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:08.331819 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:39:08.331790 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24410932_5164_4c55_adf9_dd37015fe294.slice/crio-fc2c2a08ccc38268fd7b366bccd5f821d90c037823e28f1b03640d4021e05f24 WatchSource:0}: Error finding container fc2c2a08ccc38268fd7b366bccd5f821d90c037823e28f1b03640d4021e05f24: Status 404 returned error can't find the container with id fc2c2a08ccc38268fd7b366bccd5f821d90c037823e28f1b03640d4021e05f24 Nov 03 20:39:08.331985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:08.331963 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69"] Nov 03 20:39:08.786597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:08.786557 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" event={"ID":"24410932-5164-4c55-adf9-dd37015fe294","Type":"ContainerStarted","Data":"fc2c2a08ccc38268fd7b366bccd5f821d90c037823e28f1b03640d4021e05f24"} Nov 03 20:39:09.855468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:09.855428 5815 generic.go:356] "Generic (PLEG): container finished" podID="24410932-5164-4c55-adf9-dd37015fe294" containerID="a86886beadb89b143b270d876690a00b496b6bde5dc8b7b29404aafe41d9f34d" exitCode=0 Nov 03 20:39:09.855894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:09.855534 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" event={"ID":"24410932-5164-4c55-adf9-dd37015fe294","Type":"ContainerDied","Data":"a86886beadb89b143b270d876690a00b496b6bde5dc8b7b29404aafe41d9f34d"} Nov 03 20:39:11.050422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.050399 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:11.149422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.149346 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn\" (UniqueName: \"kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn\") pod \"24410932-5164-4c55-adf9-dd37015fe294\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " Nov 03 20:39:11.149422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.149409 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca\") pod \"24410932-5164-4c55-adf9-dd37015fe294\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " Nov 03 20:39:11.149725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.149460 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca\") pod \"24410932-5164-4c55-adf9-dd37015fe294\" (UID: \"24410932-5164-4c55-adf9-dd37015fe294\") " Nov 03 20:39:11.149872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.149841 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca" pod "24410932-5164-4c55-adf9-dd37015fe294" (UID: "24410932-5164-4c55-adf9-dd37015fe294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:11.150395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.150371 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24410932-5164-4c55-adf9-dd37015fe294-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:11.152013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.151979 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn" pod "24410932-5164-4c55-adf9-dd37015fe294" (UID: "24410932-5164-4c55-adf9-dd37015fe294"). InnerVolumeSpecName "kube-api-access-s5xgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:11.152108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.152014 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca" pod "24410932-5164-4c55-adf9-dd37015fe294" (UID: "24410932-5164-4c55-adf9-dd37015fe294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:11.252016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.251986 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xgn\" (UniqueName: \"kubernetes.io/projected/24410932-5164-4c55-adf9-dd37015fe294-kube-api-access-s5xgn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:11.252016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.252014 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24410932-5164-4c55-adf9-dd37015fe294-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:11.877294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.877266 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" Nov 03 20:39:11.877528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.877309 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69" event={"ID":"24410932-5164-4c55-adf9-dd37015fe294","Type":"ContainerDied","Data":"fc2c2a08ccc38268fd7b366bccd5f821d90c037823e28f1b03640d4021e05f24"} Nov 03 20:39:11.877528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:11.877367 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2c2a08ccc38268fd7b366bccd5f821d90c037823e28f1b03640d4021e05f24" Nov 03 20:39:13.853901 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:13.853865 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:39:17.617643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:17.617609 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:39:19.117584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:19.067302 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q"] Nov 03 20:39:21.644992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.644957 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg"] Nov 03 20:39:21.645792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.645774 5815 cpu_manager.go:414] "containerMap: removing container" podUID="24410932-5164-4c55-adf9-dd37015fe294" containerName="revoke-keys" Nov 03 20:39:21.645847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.645794 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24410932-5164-4c55-adf9-dd37015fe294" containerName="revoke-keys" Nov 03 20:39:21.645958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.645949 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="24410932-5164-4c55-adf9-dd37015fe294" containerName="revoke-keys" Nov 03 20:39:21.651300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.651273 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.659902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.659876 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg"] Nov 03 20:39:21.660273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.660241 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"agc-cred-d8f519bce35f\"" Nov 03 20:39:21.825636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.814707 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.825636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.814791 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.825636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.814851 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.825636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.814934 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccx5s\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.919077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.918983 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.919792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.919765 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.927536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.923910 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.928456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.928431 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.928774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.928755 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccx5s\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.929693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.929667 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.959491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.958994 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:21.981142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:21.981071 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccx5s\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:22.273180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:22.272838 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:22.863046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:22.863002 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg"] Nov 03 20:39:22.953587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:22.953496 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s"] Nov 03 20:39:22.973561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:22.968826 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s"] Nov 03 20:39:22.973561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:22.968946 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:22.976851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:22.976829 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-b599b816968c\"" Nov 03 20:39:23.038610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.038583 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5jp\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.038862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.038846 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.039073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.039060 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.039235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.039219 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.158617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.156763 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5jp\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.158617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.156865 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.158617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.156955 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.158617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.157040 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.158617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.158574 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.161536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.159467 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.176170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.176139 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.183223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.183166 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5jp\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.333240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.333208 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:23.735093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:23.735058 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s"] Nov 03 20:39:24.142994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.142883 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" podUID="0d4bda77-152e-4ad9-83f8-2339830e9695" containerName="payload-processing" containerID="cri-o://32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e" gracePeriod=25 Nov 03 20:39:24.177579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.165805 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" event={"ID":"f6c7932c-d27f-48d3-92da-8bc486fa3fa4","Type":"ContainerStarted","Data":"33169cd9dc5986cdb00dcb139f4630b34db62426751f5c3b740d85e32d762c9b"} Nov 03 20:39:24.178008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.177982 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:24.178144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.178120 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" event={"ID":"f6c7932c-d27f-48d3-92da-8bc486fa3fa4","Type":"ContainerStarted","Data":"ea185a7b3b483e4571383b5553beadb9bf184cb64173c1af2d676717c03519ee"} Nov 03 20:39:24.289030 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:24.246277 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-db5198/agc-cred-d8f519bce35f: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:24.289030 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:24.246305 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:24.289030 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:24.246384 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials podName:f6c7932c-d27f-48d3-92da-8bc486fa3fa4 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:24.74635874 +0000 UTC m=+2274.276704406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4") : secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:24.378701 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:39:24.378611 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3143170_0524_498e_8252_306a7b49bbe2.slice/crio-db0756c3b9f6e3bd7726e281f0619db7673ab9ad2b8e97db069abdae1af55937 WatchSource:0}: Error finding container db0756c3b9f6e3bd7726e281f0619db7673ab9ad2b8e97db069abdae1af55937: Status 404 returned error can't find the container with id db0756c3b9f6e3bd7726e281f0619db7673ab9ad2b8e97db069abdae1af55937 Nov 03 20:39:24.391526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.390234 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d"] Nov 03 20:39:24.399170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.399147 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.405039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.405018 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"e2e-cfl-4557dc-provider-cred\"" Nov 03 20:39:24.405346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.405326 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-e2e--e92303d8-dockercfg-kk2d8\"" Nov 03 20:39:24.444069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.443922 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d"] Nov 03 20:39:24.444069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.443995 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" podStartSLOduration=3.443977002 podStartE2EDuration="3.443977002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:24.235419094 +0000 UTC m=+2273.765764768" watchObservedRunningTime="2026-11-03 20:39:24.443977002 +0000 UTC m=+2273.974322674" Nov 03 20:39:24.461846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.461770 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s"] Nov 03 20:39:24.475477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.474749 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5cq\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.475477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.474850 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.475477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.474967 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.475477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.475104 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.579841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.578020 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.579841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.578128 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.579841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.578232 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.579841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.578367 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5cq\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.579841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.579166 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.586023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.580360 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.586023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.582749 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.607665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.607626 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5cq\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.635465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.635446 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:39:24.680346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.680261 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials\") pod \"0d4bda77-152e-4ad9-83f8-2339830e9695\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " Nov 03 20:39:24.680722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.680699 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb\") pod \"0d4bda77-152e-4ad9-83f8-2339830e9695\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " Nov 03 20:39:24.681027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.681009 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume\") pod \"0d4bda77-152e-4ad9-83f8-2339830e9695\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " Nov 03 20:39:24.681685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.681560 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume" pod "0d4bda77-152e-4ad9-83f8-2339830e9695" (UID: "0d4bda77-152e-4ad9-83f8-2339830e9695"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:24.682223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.682189 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume" pod "0d4bda77-152e-4ad9-83f8-2339830e9695" (UID: "0d4bda77-152e-4ad9-83f8-2339830e9695"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:24.682341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.681630 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume\") pod \"0d4bda77-152e-4ad9-83f8-2339830e9695\" (UID: \"0d4bda77-152e-4ad9-83f8-2339830e9695\") " Nov 03 20:39:24.683347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.683323 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.683483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.683467 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d4bda77-152e-4ad9-83f8-2339830e9695-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.685020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.684985 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials" pod "0d4bda77-152e-4ad9-83f8-2339830e9695" (UID: "0d4bda77-152e-4ad9-83f8-2339830e9695"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:24.686910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.686885 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb" pod "0d4bda77-152e-4ad9-83f8-2339830e9695" (UID: "0d4bda77-152e-4ad9-83f8-2339830e9695"). InnerVolumeSpecName "kube-api-access-zbldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:24.751742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.751715 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:24.822496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.785915 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.822496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:24.785956 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbldb\" (UniqueName: \"kubernetes.io/projected/0d4bda77-152e-4ad9-83f8-2339830e9695-kube-api-access-zbldb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:24.822496 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:24.786079 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-db5198/agc-cred-d8f519bce35f: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:24.822496 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:24.786093 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:24.822496 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:24.786156 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials podName:f6c7932c-d27f-48d3-92da-8bc486fa3fa4 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:25.786136983 +0000 UTC m=+2275.316482648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4") : secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:25.108979 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:39:25.108948 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9555cb9_159a_4784_b6f9_7778ba070ec5.slice/crio-6c5308b3751bd4b77a3a170b82ecf0f4b59d3bd815f6e4e5906ad70a0a1b712a WatchSource:0}: Error finding container 6c5308b3751bd4b77a3a170b82ecf0f4b59d3bd815f6e4e5906ad70a0a1b712a: Status 404 returned error can't find the container with id 6c5308b3751bd4b77a3a170b82ecf0f4b59d3bd815f6e4e5906ad70a0a1b712a Nov 03 20:39:25.111175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.110810 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d"] Nov 03 20:39:25.179501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.179467 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" event={"ID":"c9555cb9-159a-4784-b6f9-7778ba070ec5","Type":"ContainerStarted","Data":"6c5308b3751bd4b77a3a170b82ecf0f4b59d3bd815f6e4e5906ad70a0a1b712a"} Nov 03 20:39:25.181468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.181432 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" event={"ID":"b3143170-0524-498e-8252-306a7b49bbe2","Type":"ContainerStarted","Data":"01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a"} Nov 03 20:39:25.181630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.181488 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" event={"ID":"b3143170-0524-498e-8252-306a7b49bbe2","Type":"ContainerStarted","Data":"db0756c3b9f6e3bd7726e281f0619db7673ab9ad2b8e97db069abdae1af55937"} Nov 03 20:39:25.181630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.181550 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:25.183546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.183481 5815 generic.go:356] "Generic (PLEG): container finished" podID="0d4bda77-152e-4ad9-83f8-2339830e9695" containerID="32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e" exitCode=0 Nov 03 20:39:25.183674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.183569 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" event={"ID":"0d4bda77-152e-4ad9-83f8-2339830e9695","Type":"ContainerDied","Data":"32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e"} Nov 03 20:39:25.183674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.183587 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" Nov 03 20:39:25.183674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.183632 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q" event={"ID":"0d4bda77-152e-4ad9-83f8-2339830e9695","Type":"ContainerDied","Data":"29bf30dc288e4e01ad346c56631dc8cce0253548497ca8e422b55549f97214e9"} Nov 03 20:39:25.183867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.183696 5815 scope.go:122] "RemoveContainer" containerID="32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e" Nov 03 20:39:25.199704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.199652 5815 scope.go:122] "RemoveContainer" containerID="32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e" Nov 03 20:39:25.208120 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:25.208030 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e\": container with ID starting with 32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e not found: ID does not exist" containerID="32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e" Nov 03 20:39:25.208120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.208071 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e"} err="failed to get container status \"32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e\": rpc error: code = NotFound desc = could not find container \"32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e\": container with ID starting with 32808ccf36ec85c415a708deff4ae668877bea1535ffee920972752fda389f6e not found: ID does not exist" Nov 03 20:39:25.208774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.208458 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" podStartSLOduration=3.208439833 podStartE2EDuration="3.208439833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:25.20664153 +0000 UTC m=+2274.736987215" watchObservedRunningTime="2026-11-03 20:39:25.208439833 +0000 UTC m=+2274.738785506" Nov 03 20:39:25.225790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.225761 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q"] Nov 03 20:39:25.230212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:25.230186 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6dffqfl9q"] Nov 03 20:39:25.829255 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:25.829226 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-db5198/agc-cred-d8f519bce35f: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:25.829255 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:25.829249 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:25.829480 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:25.829299 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials podName:f6c7932c-d27f-48d3-92da-8bc486fa3fa4 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:27.829285509 +0000 UTC m=+2277.359631172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4") : secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:26.197410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:26.197313 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" event={"ID":"c9555cb9-159a-4784-b6f9-7778ba070ec5","Type":"ContainerStarted","Data":"b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe"} Nov 03 20:39:26.197410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:26.197387 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:26.217178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:26.217133 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" podStartSLOduration=2.217118658 podStartE2EDuration="2.217118658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:26.216221857 +0000 UTC m=+2275.746567529" watchObservedRunningTime="2026-11-03 20:39:26.217118658 +0000 UTC m=+2275.747464330" Nov 03 20:39:26.588132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:26.588097 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4bda77-152e-4ad9-83f8-2339830e9695" path="/var/lib/kubelet/pods/0d4bda77-152e-4ad9-83f8-2339830e9695/volumes" Nov 03 20:39:27.878439 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:27.878404 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-db5198/agc-cred-d8f519bce35f: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:27.878439 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:27.878435 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:27.878959 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:27.878495 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials podName:f6c7932c-d27f-48d3-92da-8bc486fa3fa4 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:31.878479753 +0000 UTC m=+2281.408825419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4") : secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:30.182478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.182388 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" podUID="b3143170-0524-498e-8252-306a7b49bbe2" containerName="payload-processing" containerID="cri-o://01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a" gracePeriod=25 Nov 03 20:39:30.513234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.513206 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:30.564542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.563140 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials\") pod \"b3143170-0524-498e-8252-306a7b49bbe2\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " Nov 03 20:39:30.564542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.563453 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume\") pod \"b3143170-0524-498e-8252-306a7b49bbe2\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " Nov 03 20:39:30.564542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.563585 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume\") pod \"b3143170-0524-498e-8252-306a7b49bbe2\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " Nov 03 20:39:30.564542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.563628 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp\") pod \"b3143170-0524-498e-8252-306a7b49bbe2\" (UID: \"b3143170-0524-498e-8252-306a7b49bbe2\") " Nov 03 20:39:30.595588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.585312 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume" pod "b3143170-0524-498e-8252-306a7b49bbe2" (UID: "b3143170-0524-498e-8252-306a7b49bbe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:30.595588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.586702 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume" pod "b3143170-0524-498e-8252-306a7b49bbe2" (UID: "b3143170-0524-498e-8252-306a7b49bbe2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:30.641488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.641435 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp" pod "b3143170-0524-498e-8252-306a7b49bbe2" (UID: "b3143170-0524-498e-8252-306a7b49bbe2"). InnerVolumeSpecName "kube-api-access-kp5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:30.641686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.641584 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials" pod "b3143170-0524-498e-8252-306a7b49bbe2" (UID: "b3143170-0524-498e-8252-306a7b49bbe2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:30.671043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.670858 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:30.671043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.670898 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5jp\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-kube-api-access-kp5jp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:30.671043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.670914 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3143170-0524-498e-8252-306a7b49bbe2-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:30.671043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:30.670931 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3143170-0524-498e-8252-306a7b49bbe2-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:31.297530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.294154 5815 generic.go:356] "Generic (PLEG): container finished" podID="b3143170-0524-498e-8252-306a7b49bbe2" containerID="01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a" exitCode=0 Nov 03 20:39:31.297530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.294399 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" event={"ID":"b3143170-0524-498e-8252-306a7b49bbe2","Type":"ContainerDied","Data":"01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a"} Nov 03 20:39:31.297530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.294448 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" event={"ID":"b3143170-0524-498e-8252-306a7b49bbe2","Type":"ContainerDied","Data":"db0756c3b9f6e3bd7726e281f0619db7673ab9ad2b8e97db069abdae1af55937"} Nov 03 20:39:31.297530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.294474 5815 scope.go:122] "RemoveContainer" containerID="01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a" Nov 03 20:39:31.297530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.294824 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s" Nov 03 20:39:31.318915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.318891 5815 scope.go:122] "RemoveContainer" containerID="01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a" Nov 03 20:39:31.319278 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:31.319253 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a\": container with ID starting with 01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a not found: ID does not exist" containerID="01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a" Nov 03 20:39:31.319362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.319290 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a"} err="failed to get container status \"01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a\": rpc error: code = NotFound desc = could not find container \"01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a\": container with ID starting with 01ed5164b9fb64e042ba3409c9f58fd3266e4820e7351aaa76fce7699f79fb8a not found: ID does not exist" Nov 03 20:39:31.330605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.330577 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s"] Nov 03 20:39:31.335026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:31.335002 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4v2x4s"] Nov 03 20:39:31.897619 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:31.897576 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-db5198/agc-cred-d8f519bce35f: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:31.897619 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:31.897618 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:31.897906 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:31.897684 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials podName:f6c7932c-d27f-48d3-92da-8bc486fa3fa4 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:39.89766515 +0000 UTC m=+2289.428010802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4") : secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:32.589968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:32.589927 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3143170-0524-498e-8252-306a7b49bbe2" path="/var/lib/kubelet/pods/b3143170-0524-498e-8252-306a7b49bbe2/volumes" Nov 03 20:39:35.187561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:35.187530 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:36.357083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:36.357051 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf"] Nov 03 20:39:37.214857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:37.214825 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:39:37.276021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:37.275992 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg"] Nov 03 20:39:39.970533 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:39.969560 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-db5198/agc-cred-d8f519bce35f: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:39.970533 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:39.969594 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg: secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:39.970533 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:39.969660 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials podName:f6c7932c-d27f-48d3-92da-8bc486fa3fa4 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:55.969639868 +0000 UTC m=+2305.499985519 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4") : secret "agc-cred-d8f519bce35f" not found Nov 03 20:39:41.357823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.357780 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" podUID="42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" containerName="payload-processing" containerID="cri-o://695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270" gracePeriod=25 Nov 03 20:39:41.705842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.705801 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:39:41.813033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.812998 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials\") pod \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " Nov 03 20:39:41.813033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.813039 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume\") pod \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " Nov 03 20:39:41.813296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.813073 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume\") pod \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " Nov 03 20:39:41.813296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.813188 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg\") pod \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\" (UID: \"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb\") " Nov 03 20:39:41.813546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.813489 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume" pod "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" (UID: "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:41.813546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.813496 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume" pod "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" (UID: "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:41.814056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.814032 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.814167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.814063 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.815787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.815763 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg" pod "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" (UID: "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb"). InnerVolumeSpecName "kube-api-access-swmgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:41.815890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.815851 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials" pod "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" (UID: "42d32ff7-7f4b-43fb-8082-fdc26b2e36eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:41.915990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.915956 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:41.915990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:41.915986 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmgg\" (UniqueName: \"kubernetes.io/projected/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb-kube-api-access-swmgg\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:42.276321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.276281 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" podUID="f6c7932c-d27f-48d3-92da-8bc486fa3fa4" containerName="payload-processing" containerID="cri-o://33169cd9dc5986cdb00dcb139f4630b34db62426751f5c3b740d85e32d762c9b" gracePeriod=25 Nov 03 20:39:42.512762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.512710 5815 generic.go:356] "Generic (PLEG): container finished" podID="42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" containerID="695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270" exitCode=0 Nov 03 20:39:42.513377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.512766 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" Nov 03 20:39:42.513851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.512759 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" event={"ID":"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb","Type":"ContainerDied","Data":"695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270"} Nov 03 20:39:42.513989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.513943 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf" event={"ID":"42d32ff7-7f4b-43fb-8082-fdc26b2e36eb","Type":"ContainerDied","Data":"0ef092ccde4fa19360260cfe049b649f78c6d0e375fe217d4788b1308a9fe034"} Nov 03 20:39:42.513989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.513980 5815 scope.go:122] "RemoveContainer" containerID="695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270" Nov 03 20:39:42.519911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.519886 5815 generic.go:356] "Generic (PLEG): container finished" podID="f6c7932c-d27f-48d3-92da-8bc486fa3fa4" containerID="33169cd9dc5986cdb00dcb139f4630b34db62426751f5c3b740d85e32d762c9b" exitCode=0 Nov 03 20:39:42.520119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.520094 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" event={"ID":"f6c7932c-d27f-48d3-92da-8bc486fa3fa4","Type":"ContainerDied","Data":"33169cd9dc5986cdb00dcb139f4630b34db62426751f5c3b740d85e32d762c9b"} Nov 03 20:39:42.534959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.534939 5815 scope.go:122] "RemoveContainer" containerID="695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270" Nov 03 20:39:42.535294 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:42.535271 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270\": container with ID starting with 695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270 not found: ID does not exist" containerID="695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270" Nov 03 20:39:42.535381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.535305 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270"} err="failed to get container status \"695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270\": rpc error: code = NotFound desc = could not find container \"695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270\": container with ID starting with 695909ef8406a1125df6646427a6781771b877f65480b0f98cb2837e26512270 not found: ID does not exist" Nov 03 20:39:42.544392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.544366 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf"] Nov 03 20:39:42.549096 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.549073 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d8zdsgf"] Nov 03 20:39:42.577200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.577169 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:42.588334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.588307 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" path="/var/lib/kubelet/pods/42d32ff7-7f4b-43fb-8082-fdc26b2e36eb/volumes" Nov 03 20:39:42.626731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.626706 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume\") pod \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " Nov 03 20:39:42.626884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.626790 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume\") pod \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " Nov 03 20:39:42.626884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.626833 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s\") pod \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " Nov 03 20:39:42.627044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.627032 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials\") pod \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\" (UID: \"f6c7932c-d27f-48d3-92da-8bc486fa3fa4\") " Nov 03 20:39:42.627141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.627119 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume" pod "f6c7932c-d27f-48d3-92da-8bc486fa3fa4" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:42.627275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.627246 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume" pod "f6c7932c-d27f-48d3-92da-8bc486fa3fa4" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:42.627839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.627817 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:42.627955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.627843 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:42.629959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.629932 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s" pod "f6c7932c-d27f-48d3-92da-8bc486fa3fa4" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4"). InnerVolumeSpecName "kube-api-access-ccx5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:42.630052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.629963 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials" pod "f6c7932c-d27f-48d3-92da-8bc486fa3fa4" (UID: "f6c7932c-d27f-48d3-92da-8bc486fa3fa4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:42.729277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.729244 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccx5s\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-kube-api-access-ccx5s\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:42.729277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:42.729275 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6c7932c-d27f-48d3-92da-8bc486fa3fa4-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:43.609356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:43.604782 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" event={"ID":"f6c7932c-d27f-48d3-92da-8bc486fa3fa4","Type":"ContainerDied","Data":"ea185a7b3b483e4571383b5553beadb9bf184cb64173c1af2d676717c03519ee"} Nov 03 20:39:43.609356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:43.604858 5815 scope.go:122] "RemoveContainer" containerID="33169cd9dc5986cdb00dcb139f4630b34db62426751f5c3b740d85e32d762c9b" Nov 03 20:39:43.609356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:43.604979 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg" Nov 03 20:39:43.636626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:43.636030 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg"] Nov 03 20:39:43.640822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:43.640796 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dsdsfg"] Nov 03 20:39:43.853230 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:39:43.853199 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:39:44.590696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:44.590663 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c7932c-d27f-48d3-92da-8bc486fa3fa4" path="/var/lib/kubelet/pods/f6c7932c-d27f-48d3-92da-8bc486fa3fa4/volumes" Nov 03 20:39:52.789581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.789545 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh"] Nov 03 20:39:52.790333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790308 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0d4bda77-152e-4ad9-83f8-2339830e9695" containerName="payload-processing" Nov 03 20:39:52.790333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790320 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4bda77-152e-4ad9-83f8-2339830e9695" containerName="payload-processing" Nov 03 20:39:52.790453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790395 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f6c7932c-d27f-48d3-92da-8bc486fa3fa4" containerName="payload-processing" Nov 03 20:39:52.790453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790401 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c7932c-d27f-48d3-92da-8bc486fa3fa4" containerName="payload-processing" Nov 03 20:39:52.790453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790411 5815 cpu_manager.go:414] "containerMap: removing container" podUID="42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" containerName="payload-processing" Nov 03 20:39:52.790453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790415 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" containerName="payload-processing" Nov 03 20:39:52.790453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790427 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b3143170-0524-498e-8252-306a7b49bbe2" containerName="payload-processing" Nov 03 20:39:52.790453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790431 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3143170-0524-498e-8252-306a7b49bbe2" containerName="payload-processing" Nov 03 20:39:52.790732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790580 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c7932c-d27f-48d3-92da-8bc486fa3fa4" containerName="payload-processing" Nov 03 20:39:52.790732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790592 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d32ff7-7f4b-43fb-8082-fdc26b2e36eb" containerName="payload-processing" Nov 03 20:39:52.790732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790598 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3143170-0524-498e-8252-306a7b49bbe2" containerName="payload-processing" Nov 03 20:39:52.790732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.790605 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4bda77-152e-4ad9-83f8-2339830e9695" containerName="payload-processing" Nov 03 20:39:52.794166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.794144 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:52.801607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.800490 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:39:52.807536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.802583 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:39:52.834120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.834089 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh"] Nov 03 20:39:52.969806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.969762 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:52.970028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.969900 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8jh\" (UniqueName: \"kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:52.970097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:52.970086 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.072392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.072284 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8jh\" (UniqueName: \"kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.072607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.072487 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.072685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.072667 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.073524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.073489 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.075484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.075456 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.084600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.084577 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8jh\" (UniqueName: \"kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh\") pod \"maas-api-revoke-sub-089907d9-jvvnh\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.134998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.134975 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:53.420688 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:39:53.420658 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab487ad_1ccf_4a5e_80e8_e480f3273908.slice/crio-9037d5aef97268b4bb791a0b1e14976215e41de245b16a1b6244f2a541e5ff4a WatchSource:0}: Error finding container 9037d5aef97268b4bb791a0b1e14976215e41de245b16a1b6244f2a541e5ff4a: Status 404 returned error can't find the container with id 9037d5aef97268b4bb791a0b1e14976215e41de245b16a1b6244f2a541e5ff4a Nov 03 20:39:53.420858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.420714 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh"] Nov 03 20:39:53.817047 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:53.817008 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" event={"ID":"4ab487ad-1ccf-4a5e-80e8-e480f3273908","Type":"ContainerStarted","Data":"9037d5aef97268b4bb791a0b1e14976215e41de245b16a1b6244f2a541e5ff4a"} Nov 03 20:39:54.049633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.049568 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8"] Nov 03 20:39:54.193531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.187675 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8"] Nov 03 20:39:54.193531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.187822 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.382647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.382394 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.383060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.383034 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.383279 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.383260 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.383633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.383610 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.487828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.487798 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.488000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.487871 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.488000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.487988 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.488093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.488032 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.488709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.488682 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.488847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.488795 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.490567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.490542 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.500368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.500346 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.688121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.688065 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:54.834596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.834555 5815 generic.go:356] "Generic (PLEG): container finished" podID="4ab487ad-1ccf-4a5e-80e8-e480f3273908" containerID="0eab0992f8d92c24db4a5328031bd4dcbb4f7013d41d5b667a4d84db27820779" exitCode=0 Nov 03 20:39:54.835087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:54.834639 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" event={"ID":"4ab487ad-1ccf-4a5e-80e8-e480f3273908","Type":"ContainerDied","Data":"0eab0992f8d92c24db4a5328031bd4dcbb4f7013d41d5b667a4d84db27820779"} Nov 03 20:39:55.009913 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:39:55.009882 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704ef430_1c99_41df_8e95_68f50122638b.slice/crio-fb21f50ca2a4a2c83589cf8a32f2fb55b202588e1cd9b8e097f689d0b9ad961a WatchSource:0}: Error finding container fb21f50ca2a4a2c83589cf8a32f2fb55b202588e1cd9b8e097f689d0b9ad961a: Status 404 returned error can't find the container with id fb21f50ca2a4a2c83589cf8a32f2fb55b202588e1cd9b8e097f689d0b9ad961a Nov 03 20:39:55.012693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:55.012665 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8"] Nov 03 20:39:55.845720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:55.845689 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" event={"ID":"704ef430-1c99-41df-8e95-68f50122638b","Type":"ContainerStarted","Data":"912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e"} Nov 03 20:39:55.846243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:55.845740 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" event={"ID":"704ef430-1c99-41df-8e95-68f50122638b","Type":"ContainerStarted","Data":"fb21f50ca2a4a2c83589cf8a32f2fb55b202588e1cd9b8e097f689d0b9ad961a"} Nov 03 20:39:55.846243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:55.845795 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:39:55.866189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:55.866139 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" podStartSLOduration=1.866123368 podStartE2EDuration="1.866123368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:55.864785801 +0000 UTC m=+2305.395131474" watchObservedRunningTime="2026-11-03 20:39:55.866123368 +0000 UTC m=+2305.396469043" Nov 03 20:39:56.026053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.026028 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:56.252256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.252223 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh\" (UniqueName: \"kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh\") pod \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " Nov 03 20:39:56.252457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.252381 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca\") pod \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " Nov 03 20:39:56.252623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.252600 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca\") pod \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\" (UID: \"4ab487ad-1ccf-4a5e-80e8-e480f3273908\") " Nov 03 20:39:56.252944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.252922 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca" pod "4ab487ad-1ccf-4a5e-80e8-e480f3273908" (UID: "4ab487ad-1ccf-4a5e-80e8-e480f3273908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:39:56.253312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.253292 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:56.254584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.254561 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca" pod "4ab487ad-1ccf-4a5e-80e8-e480f3273908" (UID: "4ab487ad-1ccf-4a5e-80e8-e480f3273908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:39:56.254735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.254720 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh" pod "4ab487ad-1ccf-4a5e-80e8-e480f3273908" (UID: "4ab487ad-1ccf-4a5e-80e8-e480f3273908"). InnerVolumeSpecName "kube-api-access-zm8jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:39:56.355165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.355130 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab487ad-1ccf-4a5e-80e8-e480f3273908-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:56.355165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.355166 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8jh\" (UniqueName: \"kubernetes.io/projected/4ab487ad-1ccf-4a5e-80e8-e480f3273908-kube-api-access-zm8jh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:39:56.911358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.911276 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" Nov 03 20:39:56.911882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.911394 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh" event={"ID":"4ab487ad-1ccf-4a5e-80e8-e480f3273908","Type":"ContainerDied","Data":"9037d5aef97268b4bb791a0b1e14976215e41de245b16a1b6244f2a541e5ff4a"} Nov 03 20:39:56.911882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:39:56.911435 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9037d5aef97268b4bb791a0b1e14976215e41de245b16a1b6244f2a541e5ff4a" Nov 03 20:40:01.787806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.787775 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r"] Nov 03 20:40:01.788841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.788817 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4ab487ad-1ccf-4a5e-80e8-e480f3273908" containerName="revoke-keys" Nov 03 20:40:01.788841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.788840 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab487ad-1ccf-4a5e-80e8-e480f3273908" containerName="revoke-keys" Nov 03 20:40:01.789069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.789054 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab487ad-1ccf-4a5e-80e8-e480f3273908" containerName="revoke-keys" Nov 03 20:40:01.793290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.793269 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:01.802447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.802267 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:40:01.805553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.805479 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:01.908072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.908016 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvbq\" (UniqueName: \"kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:01.912780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.912752 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:01.912997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.912980 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:01.924067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:01.924036 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r"] Nov 03 20:40:02.018175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.018143 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.018175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.018177 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.018399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.018244 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvbq\" (UniqueName: \"kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.018855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.018832 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.020788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.020769 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.030884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.030861 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvbq\" (UniqueName: \"kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq\") pod \"maas-api-revoke-sub-4ad09fec-9j99r\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.166983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.166886 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:02.456194 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:02.456158 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6037f8_099b_4f35_a143_1097f98e08f0.slice/crio-f9971704a35c60656515b4615fe6223f0c112cd0c1a88d76abb6b566399834c1 WatchSource:0}: Error finding container f9971704a35c60656515b4615fe6223f0c112cd0c1a88d76abb6b566399834c1: Status 404 returned error can't find the container with id f9971704a35c60656515b4615fe6223f0c112cd0c1a88d76abb6b566399834c1 Nov 03 20:40:02.457069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.457040 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r"] Nov 03 20:40:02.997289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.997248 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" event={"ID":"2f6037f8-099b-4f35-a143-1097f98e08f0","Type":"ContainerStarted","Data":"d97d8bf40e947b73943df8eb3d394d40747c7bfbc6de846f5a029775b11049a2"} Nov 03 20:40:02.997845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:02.997312 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" event={"ID":"2f6037f8-099b-4f35-a143-1097f98e08f0","Type":"ContainerStarted","Data":"f9971704a35c60656515b4615fe6223f0c112cd0c1a88d76abb6b566399834c1"} Nov 03 20:40:03.029399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:03.029345 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" podStartSLOduration=2.029330818 podStartE2EDuration="2.029330818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:03.02292656 +0000 UTC m=+2312.553272236" watchObservedRunningTime="2026-11-03 20:40:03.029330818 +0000 UTC m=+2312.559676489" Nov 03 20:40:03.806434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:03.802616 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f"] Nov 03 20:40:04.057588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:04.056864 5815 generic.go:356] "Generic (PLEG): container finished" podID="2f6037f8-099b-4f35-a143-1097f98e08f0" containerID="d97d8bf40e947b73943df8eb3d394d40747c7bfbc6de846f5a029775b11049a2" exitCode=0 Nov 03 20:40:04.057588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:04.057240 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" event={"ID":"2f6037f8-099b-4f35-a143-1097f98e08f0","Type":"ContainerDied","Data":"d97d8bf40e947b73943df8eb3d394d40747c7bfbc6de846f5a029775b11049a2"} Nov 03 20:40:05.314535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.309256 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:05.349325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.349291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca\") pod \"2f6037f8-099b-4f35-a143-1097f98e08f0\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " Nov 03 20:40:05.349567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.349483 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca\") pod \"2f6037f8-099b-4f35-a143-1097f98e08f0\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " Nov 03 20:40:05.349672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.349627 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq\" (UniqueName: \"kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq\") pod \"2f6037f8-099b-4f35-a143-1097f98e08f0\" (UID: \"2f6037f8-099b-4f35-a143-1097f98e08f0\") " Nov 03 20:40:05.349784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.349759 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca" pod "2f6037f8-099b-4f35-a143-1097f98e08f0" (UID: "2f6037f8-099b-4f35-a143-1097f98e08f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:05.350460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.350436 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.360629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.360564 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca" pod "2f6037f8-099b-4f35-a143-1097f98e08f0" (UID: "2f6037f8-099b-4f35-a143-1097f98e08f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:05.362825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.362787 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq" pod "2f6037f8-099b-4f35-a143-1097f98e08f0" (UID: "2f6037f8-099b-4f35-a143-1097f98e08f0"). InnerVolumeSpecName "kube-api-access-2tvbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:05.458530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.452658 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6037f8-099b-4f35-a143-1097f98e08f0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:05.458530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:05.452699 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvbq\" (UniqueName: \"kubernetes.io/projected/2f6037f8-099b-4f35-a143-1097f98e08f0-kube-api-access-2tvbq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:06.058198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.058166 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9"] Nov 03 20:40:06.058946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.058932 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2f6037f8-099b-4f35-a143-1097f98e08f0" containerName="revoke-keys" Nov 03 20:40:06.059004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.058949 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6037f8-099b-4f35-a143-1097f98e08f0" containerName="revoke-keys" Nov 03 20:40:06.059130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.059121 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6037f8-099b-4f35-a143-1097f98e08f0" containerName="revoke-keys" Nov 03 20:40:06.062994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.062976 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.076790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.073070 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a4e63f48111b\"" Nov 03 20:40:06.076790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.073920 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2s455\"" Nov 03 20:40:06.076790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.074112 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:40:06.076790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.074162 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:40:06.076790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.074213 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:40:06.076790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.074259 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:40:06.088971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.088306 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9"] Nov 03 20:40:06.205533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.177988 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.205533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.178160 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.205533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.178243 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.205533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.178313 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9vs\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.350644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.319653 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.379539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.352801 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.379924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.379894 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9vs\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.494691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.352216 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.494691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.360547 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.494691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.460686 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.548076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.523265 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9vs\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.548076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.536521 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials\") pod \"payload-processing-external-model-6484f5959b-6ftn9\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.619145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.619046 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" Nov 03 20:40:06.687290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.687083 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r" event={"ID":"2f6037f8-099b-4f35-a143-1097f98e08f0","Type":"ContainerDied","Data":"f9971704a35c60656515b4615fe6223f0c112cd0c1a88d76abb6b566399834c1"} Nov 03 20:40:06.687290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.687147 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9971704a35c60656515b4615fe6223f0c112cd0c1a88d76abb6b566399834c1" Nov 03 20:40:06.692736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.692713 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:06.915211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.915133 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:40:06.974909 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:06.974863 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq"] Nov 03 20:40:07.008919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:07.008888 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9"] Nov 03 20:40:07.009185 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:07.009153 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a49df6_e295_4fd5_84f5_9f6cbff4b29e.slice/crio-26146c74787654a3c495ce52eddb8cf1fdf8ff9abaccc999b25c4d9acb0a6ecd WatchSource:0}: Error finding container 26146c74787654a3c495ce52eddb8cf1fdf8ff9abaccc999b25c4d9acb0a6ecd: Status 404 returned error can't find the container with id 26146c74787654a3c495ce52eddb8cf1fdf8ff9abaccc999b25c4d9acb0a6ecd Nov 03 20:40:07.630607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:07.630576 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" event={"ID":"54a49df6-e295-4fd5-84f5-9f6cbff4b29e","Type":"ContainerStarted","Data":"b0fd6552d4acdc1bf0a74725d29d87dc0c9ebbfc58bf39b7a218a29e5ab802eb"} Nov 03 20:40:07.631143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:07.630632 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" event={"ID":"54a49df6-e295-4fd5-84f5-9f6cbff4b29e","Type":"ContainerStarted","Data":"26146c74787654a3c495ce52eddb8cf1fdf8ff9abaccc999b25c4d9acb0a6ecd"} Nov 03 20:40:07.631143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:07.630681 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:07.651335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:07.651289 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" podStartSLOduration=1.651275517 podStartE2EDuration="1.651275517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:07.64930883 +0000 UTC m=+2317.179654503" watchObservedRunningTime="2026-11-03 20:40:07.651275517 +0000 UTC m=+2317.181621225" Nov 03 20:40:08.807532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:08.803652 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" podUID="26d6e20e-77cc-41bc-9b10-7b051c0111db" containerName="payload-processing" containerID="cri-o://5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93" gracePeriod=25 Nov 03 20:40:09.127562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.127536 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:40:09.170937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.170906 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84\") pod \"26d6e20e-77cc-41bc-9b10-7b051c0111db\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " Nov 03 20:40:09.171125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.171012 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials\") pod \"26d6e20e-77cc-41bc-9b10-7b051c0111db\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " Nov 03 20:40:09.171125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.171080 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume\") pod \"26d6e20e-77cc-41bc-9b10-7b051c0111db\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " Nov 03 20:40:09.171125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.171112 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume\") pod \"26d6e20e-77cc-41bc-9b10-7b051c0111db\" (UID: \"26d6e20e-77cc-41bc-9b10-7b051c0111db\") " Nov 03 20:40:09.171727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.171694 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume" pod "26d6e20e-77cc-41bc-9b10-7b051c0111db" (UID: "26d6e20e-77cc-41bc-9b10-7b051c0111db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:09.171979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.171955 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.172064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.172005 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume" pod "26d6e20e-77cc-41bc-9b10-7b051c0111db" (UID: "26d6e20e-77cc-41bc-9b10-7b051c0111db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:09.173739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.173712 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials" pod "26d6e20e-77cc-41bc-9b10-7b051c0111db" (UID: "26d6e20e-77cc-41bc-9b10-7b051c0111db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:09.173847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.173830 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84" pod "26d6e20e-77cc-41bc-9b10-7b051c0111db" (UID: "26d6e20e-77cc-41bc-9b10-7b051c0111db"). InnerVolumeSpecName "kube-api-access-6nd84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:09.273670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.273623 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.273670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.273658 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d6e20e-77cc-41bc-9b10-7b051c0111db-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.273670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.273673 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nd84\" (UniqueName: \"kubernetes.io/projected/26d6e20e-77cc-41bc-9b10-7b051c0111db-kube-api-access-6nd84\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:09.650447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.650413 5815 generic.go:356] "Generic (PLEG): container finished" podID="26d6e20e-77cc-41bc-9b10-7b051c0111db" containerID="5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93" exitCode=0 Nov 03 20:40:09.650658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.650460 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" Nov 03 20:40:09.650658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.650498 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" event={"ID":"26d6e20e-77cc-41bc-9b10-7b051c0111db","Type":"ContainerDied","Data":"5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93"} Nov 03 20:40:09.650658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.650565 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f" event={"ID":"26d6e20e-77cc-41bc-9b10-7b051c0111db","Type":"ContainerDied","Data":"72ff93c4edde2b1f61a01e268689c4d6ab65c61aa128df397ed0daad6f322966"} Nov 03 20:40:09.650658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.650589 5815 scope.go:122] "RemoveContainer" containerID="5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93" Nov 03 20:40:09.662946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.662921 5815 scope.go:122] "RemoveContainer" containerID="5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93" Nov 03 20:40:09.663223 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:40:09.663195 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93\": container with ID starting with 5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93 not found: ID does not exist" containerID="5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93" Nov 03 20:40:09.663292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.663249 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93"} err="failed to get container status \"5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93\": rpc error: code = NotFound desc = could not find container \"5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93\": container with ID starting with 5a736c0070de00a4950f56450fd8ec66c3eb47a2415c7c53cdeb8556cefaca93 not found: ID does not exist" Nov 03 20:40:09.677369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.677339 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f"] Nov 03 20:40:09.682595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.682574 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-5d9bdg69f"] Nov 03 20:40:09.801104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:09.801076 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:40:10.599978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:10.599944 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d6e20e-77cc-41bc-9b10-7b051c0111db" path="/var/lib/kubelet/pods/26d6e20e-77cc-41bc-9b10-7b051c0111db/volumes" Nov 03 20:40:11.975846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:11.975805 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" podUID="b4618239-7269-4c26-83db-bc16f831eada" containerName="payload-processing" containerID="cri-o://38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547" gracePeriod=25 Nov 03 20:40:12.306623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.306598 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:40:12.370257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.370219 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2\") pod \"b4618239-7269-4c26-83db-bc16f831eada\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " Nov 03 20:40:12.370426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.370414 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume\") pod \"b4618239-7269-4c26-83db-bc16f831eada\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " Nov 03 20:40:12.370563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.370459 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials\") pod \"b4618239-7269-4c26-83db-bc16f831eada\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " Nov 03 20:40:12.370563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.370537 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume\") pod \"b4618239-7269-4c26-83db-bc16f831eada\" (UID: \"b4618239-7269-4c26-83db-bc16f831eada\") " Nov 03 20:40:12.371385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.371082 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume" pod "b4618239-7269-4c26-83db-bc16f831eada" (UID: "b4618239-7269-4c26-83db-bc16f831eada"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:12.371385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.371179 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume" pod "b4618239-7269-4c26-83db-bc16f831eada" (UID: "b4618239-7269-4c26-83db-bc16f831eada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:12.371385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.371338 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:12.371385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.371361 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4618239-7269-4c26-83db-bc16f831eada-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:12.373857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.373814 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials" pod "b4618239-7269-4c26-83db-bc16f831eada" (UID: "b4618239-7269-4c26-83db-bc16f831eada"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:12.374427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.374391 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2" pod "b4618239-7269-4c26-83db-bc16f831eada" (UID: "b4618239-7269-4c26-83db-bc16f831eada"). InnerVolumeSpecName "kube-api-access-64qt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:12.473272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.473231 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:12.473272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.473263 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qt2\" (UniqueName: \"kubernetes.io/projected/b4618239-7269-4c26-83db-bc16f831eada-kube-api-access-64qt2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:12.683911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.683881 5815 generic.go:356] "Generic (PLEG): container finished" podID="b4618239-7269-4c26-83db-bc16f831eada" containerID="38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547" exitCode=0 Nov 03 20:40:12.684088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.683939 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" Nov 03 20:40:12.684088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.683965 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" event={"ID":"b4618239-7269-4c26-83db-bc16f831eada","Type":"ContainerDied","Data":"38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547"} Nov 03 20:40:12.684088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.684018 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq" event={"ID":"b4618239-7269-4c26-83db-bc16f831eada","Type":"ContainerDied","Data":"4ce3983eaa92e8ae32c6a00f10013296060722184307e0bf269c5b82a1bdaadf"} Nov 03 20:40:12.684088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.684048 5815 scope.go:122] "RemoveContainer" containerID="38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547" Nov 03 20:40:12.696141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.696110 5815 scope.go:122] "RemoveContainer" containerID="38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547" Nov 03 20:40:12.696444 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:40:12.696417 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547\": container with ID starting with 38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547 not found: ID does not exist" containerID="38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547" Nov 03 20:40:12.696584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.696454 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547"} err="failed to get container status \"38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547\": rpc error: code = NotFound desc = could not find container \"38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547\": container with ID starting with 38a5605a56002fe31672aed390cff9d10d36e13fe172ac8fc550936cd142c547 not found: ID does not exist" Nov 03 20:40:12.707593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.707570 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq"] Nov 03 20:40:12.715166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:12.715144 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7c8ckc8gq"] Nov 03 20:40:13.255539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.255483 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt"] Nov 03 20:40:13.256309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.256287 5815 cpu_manager.go:414] "containerMap: removing container" podUID="26d6e20e-77cc-41bc-9b10-7b051c0111db" containerName="payload-processing" Nov 03 20:40:13.256309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.256305 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d6e20e-77cc-41bc-9b10-7b051c0111db" containerName="payload-processing" Nov 03 20:40:13.256483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.256332 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b4618239-7269-4c26-83db-bc16f831eada" containerName="payload-processing" Nov 03 20:40:13.256483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.256337 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4618239-7269-4c26-83db-bc16f831eada" containerName="payload-processing" Nov 03 20:40:13.256483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.256477 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d6e20e-77cc-41bc-9b10-7b051c0111db" containerName="payload-processing" Nov 03 20:40:13.256605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.256489 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4618239-7269-4c26-83db-bc16f831eada" containerName="payload-processing" Nov 03 20:40:13.261696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.261677 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.266797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.266769 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:13.266968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.266900 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:40:13.270254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.270227 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt"] Nov 03 20:40:13.388399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.388368 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.388638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.388621 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9k7\" (UniqueName: \"kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.388699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.388654 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.489640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.489608 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9k7\" (UniqueName: \"kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.489841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.489651 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.489841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.489812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.490590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.490567 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.492314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.492291 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.506149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.506092 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9k7\" (UniqueName: \"kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7\") pod \"maas-api-revoke-sub-d520329e-qhjpt\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.581861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.581839 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:13.862489 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:40:13.862452 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:40:13.996634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:13.996611 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt"] Nov 03 20:40:13.997907 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:13.997878 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d46ab80_d9fb_4311_9ae8_0eb9d74a2c77.slice/crio-56f59037914136b653d91c97b134a8a150f3853713c4df7426ea2626ffd8a593 WatchSource:0}: Error finding container 56f59037914136b653d91c97b134a8a150f3853713c4df7426ea2626ffd8a593: Status 404 returned error can't find the container with id 56f59037914136b653d91c97b134a8a150f3853713c4df7426ea2626ffd8a593 Nov 03 20:40:14.591405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:14.591367 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4618239-7269-4c26-83db-bc16f831eada" path="/var/lib/kubelet/pods/b4618239-7269-4c26-83db-bc16f831eada/volumes" Nov 03 20:40:14.821703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:14.821663 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" event={"ID":"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77","Type":"ContainerStarted","Data":"76bf921cb9def7f407bd1e14adf61f814283137ab1e8cec22473f28b826a9bfc"} Nov 03 20:40:14.821900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:14.821715 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" event={"ID":"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77","Type":"ContainerStarted","Data":"56f59037914136b653d91c97b134a8a150f3853713c4df7426ea2626ffd8a593"} Nov 03 20:40:14.845486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:14.845430 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" podStartSLOduration=1.845417623 podStartE2EDuration="1.845417623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:14.842977102 +0000 UTC m=+2324.373322787" watchObservedRunningTime="2026-11-03 20:40:14.845417623 +0000 UTC m=+2324.375763295" Nov 03 20:40:15.858085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:15.855389 5815 generic.go:356] "Generic (PLEG): container finished" podID="5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" containerID="76bf921cb9def7f407bd1e14adf61f814283137ab1e8cec22473f28b826a9bfc" exitCode=0 Nov 03 20:40:15.858085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:15.855567 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" event={"ID":"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77","Type":"ContainerDied","Data":"76bf921cb9def7f407bd1e14adf61f814283137ab1e8cec22473f28b826a9bfc"} Nov 03 20:40:16.840763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.840718 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh"] Nov 03 20:40:16.845664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.845639 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:16.851069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.851043 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14ebbb\"/\"agc-cred-2561cdff034b\"" Nov 03 20:40:16.858309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.858278 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh"] Nov 03 20:40:16.950059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.949417 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:16.950059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.949574 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzq5q\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:16.950059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.949608 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:16.950059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:16.949661 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.051415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.051376 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.051622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.051548 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzq5q\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.051622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.051587 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.051757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.051644 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.052445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.052412 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.052673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.052626 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.054856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.054829 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.069575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.069547 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzq5q\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.149938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.149461 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:17.197534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.196809 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.260799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.260757 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca\") pod \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " Nov 03 20:40:17.260977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.260878 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7\" (UniqueName: \"kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7\") pod \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " Nov 03 20:40:17.260977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.260944 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca\") pod \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\" (UID: \"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77\") " Nov 03 20:40:17.262501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.262229 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca" pod "5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" (UID: "5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:17.264799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.264767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca" pod "5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" (UID: "5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:17.264906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.264796 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7" pod "5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" (UID: "5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77"). InnerVolumeSpecName "kube-api-access-db9k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:17.363557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.363527 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.363557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.363555 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9k7\" (UniqueName: \"kubernetes.io/projected/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-kube-api-access-db9k7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.363557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.363565 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:17.506957 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:17.506913 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01591ce_c3f6_431a_b70f_bf5a0dcb40c6.slice/crio-ac7102a14b965d3ec4c7a1d78d27093f8d12d11253b61e6d2051c1974eb2a06b WatchSource:0}: Error finding container ac7102a14b965d3ec4c7a1d78d27093f8d12d11253b61e6d2051c1974eb2a06b: Status 404 returned error can't find the container with id ac7102a14b965d3ec4c7a1d78d27093f8d12d11253b61e6d2051c1974eb2a06b Nov 03 20:40:17.510749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.510720 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh"] Nov 03 20:40:17.881034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.880940 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" event={"ID":"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6","Type":"ContainerStarted","Data":"31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0"} Nov 03 20:40:17.881034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.881006 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" event={"ID":"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6","Type":"ContainerStarted","Data":"ac7102a14b965d3ec4c7a1d78d27093f8d12d11253b61e6d2051c1974eb2a06b"} Nov 03 20:40:17.881567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.881037 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:17.884537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.884409 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" Nov 03 20:40:17.884537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.884420 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt" event={"ID":"5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77","Type":"ContainerDied","Data":"56f59037914136b653d91c97b134a8a150f3853713c4df7426ea2626ffd8a593"} Nov 03 20:40:17.884537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.884485 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56f59037914136b653d91c97b134a8a150f3853713c4df7426ea2626ffd8a593" Nov 03 20:40:17.956932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:17.950613 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" podStartSLOduration=1.950582566 podStartE2EDuration="1.950582566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:17.921316007 +0000 UTC m=+2327.451661677" watchObservedRunningTime="2026-11-03 20:40:17.950582566 +0000 UTC m=+2327.480928239" Nov 03 20:40:18.642921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:18.642892 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:40:23.806744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.806713 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv"] Nov 03 20:40:23.807683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.807666 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" containerName="revoke-keys" Nov 03 20:40:23.807769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.807686 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" containerName="revoke-keys" Nov 03 20:40:23.807839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.807829 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" containerName="revoke-keys" Nov 03 20:40:23.811756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.811739 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:23.817381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.817321 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:23.817648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.817615 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:40:23.820815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.820789 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv"] Nov 03 20:40:23.952337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.952299 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzq6\" (UniqueName: \"kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:23.952581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.952388 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:23.952680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:23.952648 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.054738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.054700 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.054908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.054894 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzq6\" (UniqueName: \"kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.055001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.054986 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.055615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.055588 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.057212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.057168 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.073775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.073750 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzq6\" (UniqueName: \"kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6\") pod \"maas-api-revoke-sub-37f0d73d-gbptv\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.136536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.135795 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:24.488580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.488552 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv"] Nov 03 20:40:24.983219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.983167 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" event={"ID":"2c903d0c-1aba-42bc-8c8e-b0d44cca353a","Type":"ContainerStarted","Data":"6232267ddb795c1001abe53b05930d5f57b61a4582a4027423872278b354ec70"} Nov 03 20:40:24.983792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:24.983236 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" event={"ID":"2c903d0c-1aba-42bc-8c8e-b0d44cca353a","Type":"ContainerStarted","Data":"d8f3ea7928909e9aaa601d171098220c0878b3d54e56f44ab3046a5c8739e9d2"} Nov 03 20:40:25.004545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:25.004452 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" podStartSLOduration=2.004435947 podStartE2EDuration="2.004435947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:25.003117232 +0000 UTC m=+2334.533462905" watchObservedRunningTime="2026-11-03 20:40:25.004435947 +0000 UTC m=+2334.534781618" Nov 03 20:40:25.995626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:25.995590 5815 generic.go:356] "Generic (PLEG): container finished" podID="2c903d0c-1aba-42bc-8c8e-b0d44cca353a" containerID="6232267ddb795c1001abe53b05930d5f57b61a4582a4027423872278b354ec70" exitCode=0 Nov 03 20:40:25.996347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:25.995644 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" event={"ID":"2c903d0c-1aba-42bc-8c8e-b0d44cca353a","Type":"ContainerDied","Data":"6232267ddb795c1001abe53b05930d5f57b61a4582a4027423872278b354ec70"} Nov 03 20:40:27.278938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.278872 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:27.457177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.457145 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca\") pod \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " Nov 03 20:40:27.457355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.457290 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca\") pod \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " Nov 03 20:40:27.457355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.457347 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6\" (UniqueName: \"kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6\") pod \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\" (UID: \"2c903d0c-1aba-42bc-8c8e-b0d44cca353a\") " Nov 03 20:40:27.457742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.457718 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca" pod "2c903d0c-1aba-42bc-8c8e-b0d44cca353a" (UID: "2c903d0c-1aba-42bc-8c8e-b0d44cca353a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:27.457973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.457926 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:27.459525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.459490 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca" pod "2c903d0c-1aba-42bc-8c8e-b0d44cca353a" (UID: "2c903d0c-1aba-42bc-8c8e-b0d44cca353a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:27.459784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.459760 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6" pod "2c903d0c-1aba-42bc-8c8e-b0d44cca353a" (UID: "2c903d0c-1aba-42bc-8c8e-b0d44cca353a"). InnerVolumeSpecName "kube-api-access-tfzq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:27.559157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.559097 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfzq6\" (UniqueName: \"kubernetes.io/projected/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-kube-api-access-tfzq6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:27.559157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:27.559123 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c903d0c-1aba-42bc-8c8e-b0d44cca353a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:28.019978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:28.019950 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" Nov 03 20:40:28.020183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:28.019981 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv" event={"ID":"2c903d0c-1aba-42bc-8c8e-b0d44cca353a","Type":"ContainerDied","Data":"d8f3ea7928909e9aaa601d171098220c0878b3d54e56f44ab3046a5c8739e9d2"} Nov 03 20:40:28.020183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:28.020030 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f3ea7928909e9aaa601d171098220c0878b3d54e56f44ab3046a5c8739e9d2" Nov 03 20:40:28.896814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:28.896778 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:40:28.959590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:28.959546 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d"] Nov 03 20:40:32.178538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:32.178488 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9"] Nov 03 20:40:32.179006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:32.178723 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" containerID="cri-o://9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1" gracePeriod=30 Nov 03 20:40:33.960129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:33.960094 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" podUID="8f224d64-55fc-4fd9-acf9-55a1e08569b4" containerName="payload-processing" containerID="cri-o://a6ef4f03da31bb877add8e02e3764a30fca9c2d662e571b2f35bc5fb5430d742" gracePeriod=25 Nov 03 20:40:34.122665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.122622 5815 generic.go:356] "Generic (PLEG): container finished" podID="8f224d64-55fc-4fd9-acf9-55a1e08569b4" containerID="a6ef4f03da31bb877add8e02e3764a30fca9c2d662e571b2f35bc5fb5430d742" exitCode=0 Nov 03 20:40:34.122968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.122687 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" event={"ID":"8f224d64-55fc-4fd9-acf9-55a1e08569b4","Type":"ContainerDied","Data":"a6ef4f03da31bb877add8e02e3764a30fca9c2d662e571b2f35bc5fb5430d742"} Nov 03 20:40:34.278795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.278770 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:40:34.382936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.382905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials\") pod \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " Nov 03 20:40:34.383146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.383014 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr\") pod \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " Nov 03 20:40:34.383326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.383309 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume\") pod \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " Nov 03 20:40:34.383404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.383353 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume\") pod \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\" (UID: \"8f224d64-55fc-4fd9-acf9-55a1e08569b4\") " Nov 03 20:40:34.383954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.383915 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume" pod "8f224d64-55fc-4fd9-acf9-55a1e08569b4" (UID: "8f224d64-55fc-4fd9-acf9-55a1e08569b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:34.384087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.383995 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume" pod "8f224d64-55fc-4fd9-acf9-55a1e08569b4" (UID: "8f224d64-55fc-4fd9-acf9-55a1e08569b4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:34.385386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.385363 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials" pod "8f224d64-55fc-4fd9-acf9-55a1e08569b4" (UID: "8f224d64-55fc-4fd9-acf9-55a1e08569b4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:34.385556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.385440 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr" pod "8f224d64-55fc-4fd9-acf9-55a1e08569b4" (UID: "8f224d64-55fc-4fd9-acf9-55a1e08569b4"). InnerVolumeSpecName "kube-api-access-p4vqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:34.492584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.488290 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.492584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.488333 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f224d64-55fc-4fd9-acf9-55a1e08569b4-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.492584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.488348 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:34.492584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:34.488362 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4vqr\" (UniqueName: \"kubernetes.io/projected/8f224d64-55fc-4fd9-acf9-55a1e08569b4-kube-api-access-p4vqr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:35.218567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:35.213325 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" event={"ID":"8f224d64-55fc-4fd9-acf9-55a1e08569b4","Type":"ContainerDied","Data":"9f5e152cd7f9ea7663a27ecc1e9d2841e759920597d32949f3cbf3c9a1eb8460"} Nov 03 20:40:35.218567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:35.213394 5815 scope.go:122] "RemoveContainer" containerID="a6ef4f03da31bb877add8e02e3764a30fca9c2d662e571b2f35bc5fb5430d742" Nov 03 20:40:35.218567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:35.213623 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d" Nov 03 20:40:35.238647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:35.238594 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d"] Nov 03 20:40:35.244783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:35.244760 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fshl4d"] Nov 03 20:40:36.644341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:36.644302 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f224d64-55fc-4fd9-acf9-55a1e08569b4" path="/var/lib/kubelet/pods/8f224d64-55fc-4fd9-acf9-55a1e08569b4/volumes" Nov 03 20:40:37.091500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.091468 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc"] Nov 03 20:40:37.092284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.092269 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2c903d0c-1aba-42bc-8c8e-b0d44cca353a" containerName="revoke-keys" Nov 03 20:40:37.092342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.092285 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c903d0c-1aba-42bc-8c8e-b0d44cca353a" containerName="revoke-keys" Nov 03 20:40:37.092342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.092311 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8f224d64-55fc-4fd9-acf9-55a1e08569b4" containerName="payload-processing" Nov 03 20:40:37.092342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.092316 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f224d64-55fc-4fd9-acf9-55a1e08569b4" containerName="payload-processing" Nov 03 20:40:37.092446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.092434 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c903d0c-1aba-42bc-8c8e-b0d44cca353a" containerName="revoke-keys" Nov 03 20:40:37.092481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.092446 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f224d64-55fc-4fd9-acf9-55a1e08569b4" containerName="payload-processing" Nov 03 20:40:37.097323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.097299 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.103795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.103766 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:37.104676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.104654 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:40:37.105668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.105645 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc"] Nov 03 20:40:37.140167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.140143 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlj5\" (UniqueName: \"kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.140300 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.140280 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.140369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.140324 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.241520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.241482 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.241646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.241541 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.241869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.241851 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlj5\" (UniqueName: \"kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.242068 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.242050 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.244100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.244081 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.253898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.253877 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlj5\" (UniqueName: \"kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5\") pod \"maas-api-revoke-sub-682c78a3-87frc\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.426217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.426165 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:37.488482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.487654 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:40:37.546001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.545969 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546139 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546191 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546307 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546346 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546402 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546430 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546466 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:37.546551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546533 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546565 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token\") pod \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\" (UID: \"20a20ebd-87e0-4fa7-82cb-8b5041895eaa\") " Nov 03 20:40:37.546648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546576 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:37.546943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546920 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:37.547043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.546984 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:37.547178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.547146 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:37.547308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.547264 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.547308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.547284 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.547308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.547300 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.547482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.547315 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.549059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.549036 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "kube-api-access-rwdxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:37.549805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.549776 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:37.549892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.549854 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:37.551250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.551224 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy" pod "20a20ebd-87e0-4fa7-82cb-8b5041895eaa" (UID: "20a20ebd-87e0-4fa7-82cb-8b5041895eaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:37.648609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.648490 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.648609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.648536 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.648609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.648551 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.648609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.648565 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.648609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.648579 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdxz\" (UniqueName: \"kubernetes.io/projected/20a20ebd-87e0-4fa7-82cb-8b5041895eaa-kube-api-access-rwdxz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:37.725171 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:37.721603 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e68fb13_e7a9_435a_a3c3_a403a2e3a1a5.slice/crio-1b7a76b7d2a8a67f262163b43319e41745d9c5ac14bd11c5188a5862ef35699d WatchSource:0}: Error finding container 1b7a76b7d2a8a67f262163b43319e41745d9c5ac14bd11c5188a5862ef35699d: Status 404 returned error can't find the container with id 1b7a76b7d2a8a67f262163b43319e41745d9c5ac14bd11c5188a5862ef35699d Nov 03 20:40:37.725171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:37.722597 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc"] Nov 03 20:40:38.244761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.244733 5815 generic.go:356] "Generic (PLEG): container finished" podID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerID="9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1" exitCode=0 Nov 03 20:40:38.244928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.244776 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" Nov 03 20:40:38.244928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.244809 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" event={"ID":"20a20ebd-87e0-4fa7-82cb-8b5041895eaa","Type":"ContainerDied","Data":"9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1"} Nov 03 20:40:38.244928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.244860 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9" event={"ID":"20a20ebd-87e0-4fa7-82cb-8b5041895eaa","Type":"ContainerDied","Data":"4a9807bf5b485d59af74aa1f79ce693fed27d1c924dbb7a570fd58182e75e0f2"} Nov 03 20:40:38.244928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.244886 5815 scope.go:122] "RemoveContainer" containerID="9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1" Nov 03 20:40:38.246817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.246789 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" event={"ID":"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5","Type":"ContainerStarted","Data":"d60bdb6062c8f553dbd1deefe89e9ecf3fb9ca4b73a1cc6784695983be37bd81"} Nov 03 20:40:38.246948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.246846 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" event={"ID":"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5","Type":"ContainerStarted","Data":"1b7a76b7d2a8a67f262163b43319e41745d9c5ac14bd11c5188a5862ef35699d"} Nov 03 20:40:38.256613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.256533 5815 scope.go:122] "RemoveContainer" containerID="9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1" Nov 03 20:40:38.256838 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:40:38.256819 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1\": container with ID starting with 9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1 not found: ID does not exist" containerID="9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1" Nov 03 20:40:38.256896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.256846 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1"} err="failed to get container status \"9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1\": rpc error: code = NotFound desc = could not find container \"9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1\": container with ID starting with 9205b4b1298957c12c0fae19e7cda2394ce5d6f4b81633a4e6a746edbd8c09b1 not found: ID does not exist" Nov 03 20:40:38.268624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.268580 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" podStartSLOduration=1.268567486 podStartE2EDuration="1.268567486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:38.266572635 +0000 UTC m=+2347.796918308" watchObservedRunningTime="2026-11-03 20:40:38.268567486 +0000 UTC m=+2347.798913157" Nov 03 20:40:38.287528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.285159 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9"] Nov 03 20:40:38.290488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.290468 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-rvtk9"] Nov 03 20:40:38.587954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:38.587925 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" path="/var/lib/kubelet/pods/20a20ebd-87e0-4fa7-82cb-8b5041895eaa/volumes" Nov 03 20:40:39.258462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:39.258427 5815 generic.go:356] "Generic (PLEG): container finished" podID="3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" containerID="d60bdb6062c8f553dbd1deefe89e9ecf3fb9ca4b73a1cc6784695983be37bd81" exitCode=0 Nov 03 20:40:39.258913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:39.258549 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" event={"ID":"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5","Type":"ContainerDied","Data":"d60bdb6062c8f553dbd1deefe89e9ecf3fb9ca4b73a1cc6784695983be37bd81"} Nov 03 20:40:40.452303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.452274 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:40.498113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.498080 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca\") pod \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " Nov 03 20:40:40.498284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.498125 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5\" (UniqueName: \"kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5\") pod \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " Nov 03 20:40:40.498442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.498423 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca\") pod \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\" (UID: \"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5\") " Nov 03 20:40:40.498534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.498455 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca" pod "3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" (UID: "3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:40.499339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.499314 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.514562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.509922 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5" pod "3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" (UID: "3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5"). InnerVolumeSpecName "kube-api-access-mwlj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:40.514562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.511877 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca" pod "3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" (UID: "3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:40.636675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.636573 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.636999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:40.636952 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlj5\" (UniqueName: \"kubernetes.io/projected/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5-kube-api-access-mwlj5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:41.357320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:41.357277 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" event={"ID":"3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5","Type":"ContainerDied","Data":"1b7a76b7d2a8a67f262163b43319e41745d9c5ac14bd11c5188a5862ef35699d"} Nov 03 20:40:41.357857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:41.357838 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b7a76b7d2a8a67f262163b43319e41745d9c5ac14bd11c5188a5862ef35699d" Nov 03 20:40:41.358025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:41.357770 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc" Nov 03 20:40:43.894113 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:40:43.894079 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:40:43.985792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:43.985762 5815 scope.go:122] "RemoveContainer" containerID="585a47af37123348438578daad13203f337059b38e696d8c3daf50b21920179b" Nov 03 20:40:48.084339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.084303 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj"] Nov 03 20:40:48.085131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.085114 5815 cpu_manager.go:414] "containerMap: removing container" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" Nov 03 20:40:48.085198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.085133 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" Nov 03 20:40:48.085198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.085143 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" containerName="revoke-keys" Nov 03 20:40:48.085198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.085148 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" containerName="revoke-keys" Nov 03 20:40:48.085306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.085300 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a20ebd-87e0-4fa7-82cb-8b5041895eaa" containerName="istio-proxy" Nov 03 20:40:48.085343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.085308 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" containerName="revoke-keys" Nov 03 20:40:48.090787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.090767 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.096596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.096572 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:40:48.096758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.096645 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:48.099138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.099092 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj"] Nov 03 20:40:48.173832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.173792 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.174023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.173977 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjw88\" (UniqueName: \"kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.174137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.174031 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.276053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.276022 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.276231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.276132 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjw88\" (UniqueName: \"kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.276231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.276163 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.276637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.276618 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.278907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.278886 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.288568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.288548 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjw88\" (UniqueName: \"kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88\") pod \"maas-api-revoke-sub-3aba4f5b-gmggj\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:48.417604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:48.417484 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:49.008847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.008552 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8"] Nov 03 20:40:49.017536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.017294 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj"] Nov 03 20:40:49.457900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.454061 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l"] Nov 03 20:40:49.461935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.461911 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.462220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.462199 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l"] Nov 03 20:40:49.521226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.521188 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" event={"ID":"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0","Type":"ContainerStarted","Data":"df9132ec58b788a5692010013d6ee9cc136094cf19634f977e59667b9021c45f"} Nov 03 20:40:49.521429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.521271 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" event={"ID":"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0","Type":"ContainerStarted","Data":"6be0977cbc01ba4ba11a9d68f74e2070314ef07a2bd4ea3e0345c923f1d36011"} Nov 03 20:40:49.532407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.532370 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbc5\" (UniqueName: \"kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.532575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.532431 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.532630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.532593 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.545474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.545409 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" podStartSLOduration=1.5453933850000001 podStartE2EDuration="1.545393385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:49.541490547 +0000 UTC m=+2359.071836219" watchObservedRunningTime="2026-11-03 20:40:49.545393385 +0000 UTC m=+2359.075739058" Nov 03 20:40:49.636564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.634697 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbc5\" (UniqueName: \"kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.636564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.634804 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.636564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.634912 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.640531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.638840 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.644525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.642360 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.648597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.648573 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbc5\" (UniqueName: \"kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5\") pod \"maas-api-revoke-sub-f2e370b2-qvh7l\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:49.789375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:49.789346 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:50.105981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:50.105946 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l"] Nov 03 20:40:50.106428 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:50.106402 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaae1739_220c_4432_a050_685c13cd1f30.slice/crio-936b869f5f95cbb1b56c6413547920969783b842830124d82d0fc86a44762b2b WatchSource:0}: Error finding container 936b869f5f95cbb1b56c6413547920969783b842830124d82d0fc86a44762b2b: Status 404 returned error can't find the container with id 936b869f5f95cbb1b56c6413547920969783b842830124d82d0fc86a44762b2b Nov 03 20:40:50.536063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:50.536023 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" event={"ID":"eaae1739-220c-4432-a050-685c13cd1f30","Type":"ContainerStarted","Data":"48c827816c3b8853d3a1d55b0fc99bd26e559c6954aff9439dbb1942efdd188e"} Nov 03 20:40:50.536687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:50.536097 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" event={"ID":"eaae1739-220c-4432-a050-685c13cd1f30","Type":"ContainerStarted","Data":"936b869f5f95cbb1b56c6413547920969783b842830124d82d0fc86a44762b2b"} Nov 03 20:40:50.538484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:50.538453 5815 generic.go:356] "Generic (PLEG): container finished" podID="4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" containerID="df9132ec58b788a5692010013d6ee9cc136094cf19634f977e59667b9021c45f" exitCode=0 Nov 03 20:40:50.538651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:50.538535 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" event={"ID":"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0","Type":"ContainerDied","Data":"df9132ec58b788a5692010013d6ee9cc136094cf19634f977e59667b9021c45f"} Nov 03 20:40:50.564380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:50.564318 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" podStartSLOduration=1.564300513 podStartE2EDuration="1.564300513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:50.555207135 +0000 UTC m=+2360.085552813" watchObservedRunningTime="2026-11-03 20:40:50.564300513 +0000 UTC m=+2360.094646187" Nov 03 20:40:51.549009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.548962 5815 generic.go:356] "Generic (PLEG): container finished" podID="eaae1739-220c-4432-a050-685c13cd1f30" containerID="48c827816c3b8853d3a1d55b0fc99bd26e559c6954aff9439dbb1942efdd188e" exitCode=0 Nov 03 20:40:51.549523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.549042 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" event={"ID":"eaae1739-220c-4432-a050-685c13cd1f30","Type":"ContainerDied","Data":"48c827816c3b8853d3a1d55b0fc99bd26e559c6954aff9439dbb1942efdd188e"} Nov 03 20:40:51.841100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.841033 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:51.904697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.904664 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca\") pod \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " Nov 03 20:40:51.904905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.904882 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca\") pod \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " Nov 03 20:40:51.905156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.905087 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88\" (UniqueName: \"kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88\") pod \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\" (UID: \"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0\") " Nov 03 20:40:51.905936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.905906 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca" pod "4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" (UID: "4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:51.909759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.908268 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88" pod "4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" (UID: "4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0"). InnerVolumeSpecName "kube-api-access-fjw88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:51.911268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:51.911244 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca" pod "4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" (UID: "4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:52.007252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.007222 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.007252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.007255 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjw88\" (UniqueName: \"kubernetes.io/projected/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-kube-api-access-fjw88\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.007526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.007269 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.619671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.619644 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" Nov 03 20:40:52.620920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.619893 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj" event={"ID":"4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0","Type":"ContainerDied","Data":"6be0977cbc01ba4ba11a9d68f74e2070314ef07a2bd4ea3e0345c923f1d36011"} Nov 03 20:40:52.621157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.621139 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be0977cbc01ba4ba11a9d68f74e2070314ef07a2bd4ea3e0345c923f1d36011" Nov 03 20:40:52.794219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.794190 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:52.820306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.820273 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca\") pod \"eaae1739-220c-4432-a050-685c13cd1f30\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " Nov 03 20:40:52.820487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.820433 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5\" (UniqueName: \"kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5\") pod \"eaae1739-220c-4432-a050-685c13cd1f30\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " Nov 03 20:40:52.820933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.820606 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca\") pod \"eaae1739-220c-4432-a050-685c13cd1f30\" (UID: \"eaae1739-220c-4432-a050-685c13cd1f30\") " Nov 03 20:40:52.821830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.821802 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca" pod "eaae1739-220c-4432-a050-685c13cd1f30" (UID: "eaae1739-220c-4432-a050-685c13cd1f30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:52.824600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.824031 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca" pod "eaae1739-220c-4432-a050-685c13cd1f30" (UID: "eaae1739-220c-4432-a050-685c13cd1f30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:52.824864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.824821 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5" pod "eaae1739-220c-4432-a050-685c13cd1f30" (UID: "eaae1739-220c-4432-a050-685c13cd1f30"). InnerVolumeSpecName "kube-api-access-rrbc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:52.923441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.923348 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbc5\" (UniqueName: \"kubernetes.io/projected/eaae1739-220c-4432-a050-685c13cd1f30-kube-api-access-rrbc5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.923441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.923390 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.923441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:52.923405 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaae1739-220c-4432-a050-685c13cd1f30-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.630926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.630896 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" Nov 03 20:40:53.630926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.630901 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l" event={"ID":"eaae1739-220c-4432-a050-685c13cd1f30","Type":"ContainerDied","Data":"936b869f5f95cbb1b56c6413547920969783b842830124d82d0fc86a44762b2b"} Nov 03 20:40:53.631737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.630995 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="936b869f5f95cbb1b56c6413547920969783b842830124d82d0fc86a44762b2b" Nov 03 20:40:53.785489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.785449 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd"] Nov 03 20:40:53.786665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.786645 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" containerName="revoke-keys" Nov 03 20:40:53.786726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.786667 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" containerName="revoke-keys" Nov 03 20:40:53.786726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.786685 5815 cpu_manager.go:414] "containerMap: removing container" podUID="eaae1739-220c-4432-a050-685c13cd1f30" containerName="revoke-keys" Nov 03 20:40:53.786726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.786692 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaae1739-220c-4432-a050-685c13cd1f30" containerName="revoke-keys" Nov 03 20:40:53.786896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.786883 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" containerName="revoke-keys" Nov 03 20:40:53.786935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.786903 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaae1739-220c-4432-a050-685c13cd1f30" containerName="revoke-keys" Nov 03 20:40:53.796540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.791986 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.798129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.798098 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:40:53.798351 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.798325 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:40:53.798714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.798694 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd"] Nov 03 20:40:53.844718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.844687 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w242\" (UniqueName: \"kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.844878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.844743 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.844966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.844912 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.946567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.946463 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w242\" (UniqueName: \"kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.946567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.946537 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.946810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.946609 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.947309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.947284 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.949356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.949330 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:53.968071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:53.968041 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w242\" (UniqueName: \"kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242\") pod \"maas-api-revoke-sub-6f981cd5-9fghd\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:54.017016 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.016968 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" podUID="704ef430-1c99-41df-8e95-68f50122638b" containerName="payload-processing" containerID="cri-o://912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e" gracePeriod=25 Nov 03 20:40:54.112132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.112102 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:54.410385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.410338 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd"] Nov 03 20:40:54.463225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.463185 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:40:54.563871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.563842 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8\") pod \"704ef430-1c99-41df-8e95-68f50122638b\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " Nov 03 20:40:54.564081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.563925 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume\") pod \"704ef430-1c99-41df-8e95-68f50122638b\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " Nov 03 20:40:54.564081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.563961 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials\") pod \"704ef430-1c99-41df-8e95-68f50122638b\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " Nov 03 20:40:54.564081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.564009 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume\") pod \"704ef430-1c99-41df-8e95-68f50122638b\" (UID: \"704ef430-1c99-41df-8e95-68f50122638b\") " Nov 03 20:40:54.564423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.564395 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume" pod "704ef430-1c99-41df-8e95-68f50122638b" (UID: "704ef430-1c99-41df-8e95-68f50122638b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:54.564531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.564443 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume" pod "704ef430-1c99-41df-8e95-68f50122638b" (UID: "704ef430-1c99-41df-8e95-68f50122638b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:54.564745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.564721 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:54.564745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.564746 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/704ef430-1c99-41df-8e95-68f50122638b-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:54.566332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.566292 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials" pod "704ef430-1c99-41df-8e95-68f50122638b" (UID: "704ef430-1c99-41df-8e95-68f50122638b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:54.566332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.566322 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8" pod "704ef430-1c99-41df-8e95-68f50122638b" (UID: "704ef430-1c99-41df-8e95-68f50122638b"). InnerVolumeSpecName "kube-api-access-fjvx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:54.640467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.640440 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" event={"ID":"c1f28329-b52f-4b55-9d92-f9e59f268c22","Type":"ContainerStarted","Data":"12884afe25ddd4e5e64482b8d6e915c588b754cab030f8e7e244e6f0a38d5e27"} Nov 03 20:40:54.642189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.642165 5815 generic.go:356] "Generic (PLEG): container finished" podID="704ef430-1c99-41df-8e95-68f50122638b" containerID="912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e" exitCode=0 Nov 03 20:40:54.642332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.642228 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" Nov 03 20:40:54.642332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.642236 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" event={"ID":"704ef430-1c99-41df-8e95-68f50122638b","Type":"ContainerDied","Data":"912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e"} Nov 03 20:40:54.642332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.642290 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8" event={"ID":"704ef430-1c99-41df-8e95-68f50122638b","Type":"ContainerDied","Data":"fb21f50ca2a4a2c83589cf8a32f2fb55b202588e1cd9b8e097f689d0b9ad961a"} Nov 03 20:40:54.642332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.642321 5815 scope.go:122] "RemoveContainer" containerID="912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e" Nov 03 20:40:54.657183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.657163 5815 scope.go:122] "RemoveContainer" containerID="912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e" Nov 03 20:40:54.657477 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:40:54.657447 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e\": container with ID starting with 912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e not found: ID does not exist" containerID="912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e" Nov 03 20:40:54.657564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.657485 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e"} err="failed to get container status \"912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e\": rpc error: code = NotFound desc = could not find container \"912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e\": container with ID starting with 912ae5a5fc8a00d65f03adbbf11b7337fead6f72d5c721ccf6310c9f2a5a572e not found: ID does not exist" Nov 03 20:40:54.664702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.664677 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8"] Nov 03 20:40:54.665950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.665925 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjvx8\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-kube-api-access-fjvx8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:54.666040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.665954 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/704ef430-1c99-41df-8e95-68f50122638b-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:54.669822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.669802 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-5bc9qfjl8"] Nov 03 20:40:54.909942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.909915 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s"] Nov 03 20:40:54.910703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.910686 5815 cpu_manager.go:414] "containerMap: removing container" podUID="704ef430-1c99-41df-8e95-68f50122638b" containerName="payload-processing" Nov 03 20:40:54.910757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.910705 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704ef430-1c99-41df-8e95-68f50122638b" containerName="payload-processing" Nov 03 20:40:54.910874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.910865 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="704ef430-1c99-41df-8e95-68f50122638b" containerName="payload-processing" Nov 03 20:40:54.914623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.914606 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:54.926864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.923391 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s"] Nov 03 20:40:54.970914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.970880 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg4ls\" (UniqueName: \"kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:54.971115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.971096 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:54.971223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:54.971195 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.079870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.078665 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg4ls\" (UniqueName: \"kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.079870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.078832 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.079870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.078925 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.079870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.079779 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.082327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.082262 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.094633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.094610 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg4ls\" (UniqueName: \"kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls\") pod \"maas-api-revoke-sub-c99edc7a-dzj5s\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.236158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.236132 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:55.536703 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:40:55.536673 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b61b1ff_d2fc_4ff0_96fd_cc67dc0db870.slice/crio-ad86a1028698fb867e33593b9ead34effd6d542e64f6a6f369bcc8a63bb1c189 WatchSource:0}: Error finding container ad86a1028698fb867e33593b9ead34effd6d542e64f6a6f369bcc8a63bb1c189: Status 404 returned error can't find the container with id ad86a1028698fb867e33593b9ead34effd6d542e64f6a6f369bcc8a63bb1c189 Nov 03 20:40:55.537032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.537006 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s"] Nov 03 20:40:55.696773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.696738 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" event={"ID":"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870","Type":"ContainerStarted","Data":"ad86a1028698fb867e33593b9ead34effd6d542e64f6a6f369bcc8a63bb1c189"} Nov 03 20:40:55.698524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.698483 5815 generic.go:356] "Generic (PLEG): container finished" podID="c1f28329-b52f-4b55-9d92-f9e59f268c22" containerID="a8a52edf57c8cc022d29b0cbeba78ace1b012b026ccbc8d0619d157656f69bba" exitCode=0 Nov 03 20:40:55.698663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:55.698563 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" event={"ID":"c1f28329-b52f-4b55-9d92-f9e59f268c22","Type":"ContainerDied","Data":"a8a52edf57c8cc022d29b0cbeba78ace1b012b026ccbc8d0619d157656f69bba"} Nov 03 20:40:56.620190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.620151 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704ef430-1c99-41df-8e95-68f50122638b" path="/var/lib/kubelet/pods/704ef430-1c99-41df-8e95-68f50122638b/volumes" Nov 03 20:40:56.711996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.711959 5815 generic.go:356] "Generic (PLEG): container finished" podID="1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" containerID="e011d5d3f57443d349bd5a37fd5b3bc3d240c7c466a6b45c58922f04f2f13142" exitCode=0 Nov 03 20:40:56.712483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.712042 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" event={"ID":"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870","Type":"ContainerDied","Data":"e011d5d3f57443d349bd5a37fd5b3bc3d240c7c466a6b45c58922f04f2f13142"} Nov 03 20:40:56.896570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.896541 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:56.911716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.911687 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242\" (UniqueName: \"kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242\") pod \"c1f28329-b52f-4b55-9d92-f9e59f268c22\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " Nov 03 20:40:56.911915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.911861 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca\") pod \"c1f28329-b52f-4b55-9d92-f9e59f268c22\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " Nov 03 20:40:56.912276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.912147 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca\") pod \"c1f28329-b52f-4b55-9d92-f9e59f268c22\" (UID: \"c1f28329-b52f-4b55-9d92-f9e59f268c22\") " Nov 03 20:40:56.914897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.914743 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca" pod "c1f28329-b52f-4b55-9d92-f9e59f268c22" (UID: "c1f28329-b52f-4b55-9d92-f9e59f268c22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:56.923282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.923258 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242" pod "c1f28329-b52f-4b55-9d92-f9e59f268c22" (UID: "c1f28329-b52f-4b55-9d92-f9e59f268c22"). InnerVolumeSpecName "kube-api-access-2w242". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:56.926054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:56.925967 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca" pod "c1f28329-b52f-4b55-9d92-f9e59f268c22" (UID: "c1f28329-b52f-4b55-9d92-f9e59f268c22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:57.014589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.014552 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.014589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.014591 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w242\" (UniqueName: \"kubernetes.io/projected/c1f28329-b52f-4b55-9d92-f9e59f268c22-kube-api-access-2w242\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.014918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.014610 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f28329-b52f-4b55-9d92-f9e59f268c22-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.722422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.722389 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" Nov 03 20:40:57.722884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.722427 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd" event={"ID":"c1f28329-b52f-4b55-9d92-f9e59f268c22","Type":"ContainerDied","Data":"12884afe25ddd4e5e64482b8d6e915c588b754cab030f8e7e244e6f0a38d5e27"} Nov 03 20:40:57.722884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.722473 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12884afe25ddd4e5e64482b8d6e915c588b754cab030f8e7e244e6f0a38d5e27" Nov 03 20:40:57.897435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.897398 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:57.928287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.928253 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca\") pod \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " Nov 03 20:40:57.928469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.928326 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca\") pod \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " Nov 03 20:40:57.928632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.928613 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls\" (UniqueName: \"kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls\") pod \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\" (UID: \"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870\") " Nov 03 20:40:57.928728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.928644 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca" pod "1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" (UID: "1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:57.929490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.929464 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:57.931278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.931248 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca" pod "1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" (UID: "1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:57.931400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:57.931287 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls" pod "1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" (UID: "1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870"). InnerVolumeSpecName "kube-api-access-xg4ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:58.031714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:58.031585 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg4ls\" (UniqueName: \"kubernetes.io/projected/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-kube-api-access-xg4ls\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:58.031930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:58.031904 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:58.734367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:58.734336 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" Nov 03 20:40:58.734860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:58.734362 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s" event={"ID":"1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870","Type":"ContainerDied","Data":"ad86a1028698fb867e33593b9ead34effd6d542e64f6a6f369bcc8a63bb1c189"} Nov 03 20:40:58.734860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:58.734437 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad86a1028698fb867e33593b9ead34effd6d542e64f6a6f369bcc8a63bb1c189" Nov 03 20:40:59.123120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.122102 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d"] Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.854865 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5"] Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.856183 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c1f28329-b52f-4b55-9d92-f9e59f268c22" containerName="revoke-keys" Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.856207 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f28329-b52f-4b55-9d92-f9e59f268c22" containerName="revoke-keys" Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.856233 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" containerName="revoke-keys" Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.856241 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" containerName="revoke-keys" Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.856519 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f28329-b52f-4b55-9d92-f9e59f268c22" containerName="revoke-keys" Nov 03 20:40:59.863532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.856534 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" containerName="revoke-keys" Nov 03 20:40:59.996535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:40:59.995857 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.033831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.033799 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-e2e--e92303d8-dockercfg-bsxgs\"" Nov 03 20:41:00.037671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.037643 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"agc-cred-d8f519bce35f\"" Nov 03 20:41:00.075910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.059807 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.088578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.077266 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.089170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.089141 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5br2\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.204229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.204141 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.216795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.216751 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5"] Nov 03 20:41:00.307602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.307556 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5br2\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.308384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.308352 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.308744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.308723 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.308975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.308954 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.337539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.310344 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.356869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.328273 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.357104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.355119 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.385540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.377492 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5br2\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.498924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.498890 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:00.666843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.666809 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5"] Nov 03 20:41:00.758193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.750676 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk"] Nov 03 20:41:00.816961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.816932 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk"] Nov 03 20:41:00.817243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.816991 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:00.831248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.829752 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:41:00.831248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.829858 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:41:00.976849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.962132 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:00.976849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.962221 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6ks\" (UniqueName: \"kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:00.976849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:00.962300 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.091583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.086314 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.091583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.086388 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6ks\" (UniqueName: \"kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.091583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.086453 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.109543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.108745 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.117531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.115558 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr"] Nov 03 20:41:01.130531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.128216 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.144530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.132068 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.154241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.153677 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6ks\" (UniqueName: \"kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks\") pod \"maas-api-revoke-sub-5bfc53bb-xw6vk\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.256256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.240557 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj"] Nov 03 20:41:01.262738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.258799 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.291702 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7g2j\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.291786 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.291817 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.291935 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.291994 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.292021 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.292091 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.292188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.292143 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.293902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.292229 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.310345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.310319 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr"] Nov 03 20:41:01.318173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.318151 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj"] Nov 03 20:41:01.327801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.327778 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5"] Nov 03 20:41:01.393950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.393919 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.393966 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394138 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394197 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394232 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394306 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394367 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394451 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394487 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394538 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394565 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394571 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394600 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394712 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.394958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394806 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7g2j\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.394958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.394837 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvq8\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.395239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395219 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.395337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395237 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.395337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395288 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.395337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395323 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.395552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395491 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.395768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395749 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.396010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.395992 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.397269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.397231 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.398312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.398282 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.408118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.408072 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7g2j\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.410012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.409991 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-dtftr\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.437021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.437000 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:01.496183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.495823 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:01.497114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497071 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.497259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497158 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.497259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497194 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497398 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497435 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497477 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497530 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497627 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.497666 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvq8\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.498365 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.498662 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.498891 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.499872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.499124 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.500628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.500370 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.504769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.504134 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.508071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.508045 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.511958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.511055 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvq8\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.518993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.518961 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-k6ntj\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.646274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.646195 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:01.838574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.838303 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk"] Nov 03 20:41:01.847401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.847376 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr"] Nov 03 20:41:01.848420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.848386 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:41:01.848554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.848466 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:41:01.848646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:01.848560 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:41:02.000454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.000420 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj"] Nov 03 20:41:02.031722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.031687 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" event={"ID":"d28af1a7-45a3-4ed5-895c-b8370f9d0496","Type":"ContainerStarted","Data":"757b90b50c8f0a493d1a7c94f34cd996a6a4cb831f89070bb393d5bdb8d2b9b8"} Nov 03 20:41:02.033242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.033213 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" event={"ID":"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64","Type":"ContainerStarted","Data":"63352fb38c1818b475b9ba895ab25bb92836e57cc92d9f5a12fbc5db1cb31277"} Nov 03 20:41:02.035121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.035097 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" event={"ID":"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b","Type":"ContainerStarted","Data":"a7ba373c523bc5c0f647f72328892207d93c7e3dcb65473671f9a8850deb9226"} Nov 03 20:41:02.035247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.035141 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" event={"ID":"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b","Type":"ContainerStarted","Data":"fed0cedcad88e1d8d96b453e4788a921194a3272dc508b7c4490cc1393d97321"} Nov 03 20:41:02.035247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.035191 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:02.060498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.060453 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" podStartSLOduration=3.060440223 podStartE2EDuration="3.060440223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:02.054694083 +0000 UTC m=+2371.585039754" watchObservedRunningTime="2026-11-03 20:41:02.060440223 +0000 UTC m=+2371.590785895" Nov 03 20:41:02.069978 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:02.069940 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51daf21c_856d_4f43_ba29_77e134c1b464.slice/crio-c52319752f0096e485223e1419d70e5e2d3643c60adbe9d0dfe9cbf6c7fef03f WatchSource:0}: Error finding container c52319752f0096e485223e1419d70e5e2d3643c60adbe9d0dfe9cbf6c7fef03f: Status 404 returned error can't find the container with id c52319752f0096e485223e1419d70e5e2d3643c60adbe9d0dfe9cbf6c7fef03f Nov 03 20:41:02.074416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.074384 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:41:02.074526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.074453 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:41:02.074526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:02.074487 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:41:03.054864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.054825 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" event={"ID":"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64","Type":"ContainerStarted","Data":"8d464f71112fa5a0837c28475f5f63505a0a707017e82b4e8ab594692c37a3c0"} Nov 03 20:41:03.057164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.057129 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" event={"ID":"51daf21c-856d-4f43-ba29-77e134c1b464","Type":"ContainerStarted","Data":"a91a2ed742393ea07b306dd840678d01ebf2f778ce88a0ee494cf1553823e422"} Nov 03 20:41:03.057324 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.057189 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" event={"ID":"51daf21c-856d-4f43-ba29-77e134c1b464","Type":"ContainerStarted","Data":"c52319752f0096e485223e1419d70e5e2d3643c60adbe9d0dfe9cbf6c7fef03f"} Nov 03 20:41:03.059957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.059931 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" event={"ID":"d28af1a7-45a3-4ed5-895c-b8370f9d0496","Type":"ContainerStarted","Data":"c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0"} Nov 03 20:41:03.165541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.156109 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" podStartSLOduration=3.156089555 podStartE2EDuration="3.156089555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:03.116682518 +0000 UTC m=+2372.647028215" watchObservedRunningTime="2026-11-03 20:41:03.156089555 +0000 UTC m=+2372.686435228" Nov 03 20:41:03.238686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.229060 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podStartSLOduration=2.229043379 podStartE2EDuration="2.229043379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:03.142073746 +0000 UTC m=+2372.672419426" watchObservedRunningTime="2026-11-03 20:41:03.229043379 +0000 UTC m=+2372.759389051" Nov 03 20:41:03.238686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.235987 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podStartSLOduration=2.235973468 podStartE2EDuration="2.235973468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:03.191189861 +0000 UTC m=+2372.721535533" watchObservedRunningTime="2026-11-03 20:41:03.235973468 +0000 UTC m=+2372.766319143" Nov 03 20:41:03.514549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.508364 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:03.514549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.508488 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:03.514549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.508550 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:03.647540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.647466 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:03.650414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.649551 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:03.650414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:03.649611 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:04.160532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.154664 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" podUID="c9555cb9-159a-4784-b6f9-7778ba070ec5" containerName="payload-processing" containerID="cri-o://b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe" gracePeriod=25 Nov 03 20:41:04.175530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.172167 5815 generic.go:356] "Generic (PLEG): container finished" podID="3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" containerID="8d464f71112fa5a0837c28475f5f63505a0a707017e82b4e8ab594692c37a3c0" exitCode=0 Nov 03 20:41:04.175530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.173942 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" event={"ID":"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64","Type":"ContainerDied","Data":"8d464f71112fa5a0837c28475f5f63505a0a707017e82b4e8ab594692c37a3c0"} Nov 03 20:41:04.499267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.498730 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:04.499267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.498797 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:04.647380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.646883 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:04.647380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:04.646970 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:05.148254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.148185 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:41:05.164101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.164071 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v"] Nov 03 20:41:05.213535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.165668 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c9555cb9-159a-4784-b6f9-7778ba070ec5" containerName="payload-processing" Nov 03 20:41:05.213765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.213739 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9555cb9-159a-4784-b6f9-7778ba070ec5" containerName="payload-processing" Nov 03 20:41:05.240536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.220080 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9555cb9-159a-4784-b6f9-7778ba070ec5" containerName="payload-processing" Nov 03 20:41:05.248332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.248308 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.309462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.291082 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d8d0fa\"/\"routing-overlay\"" Nov 03 20:41:05.309462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.291959 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d8d0fa\"/\"payload-processing-external-model-e2e--59b11050-dockercfg-d8txz\"" Nov 03 20:41:05.309462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.292047 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d8d0fa\"/\"agc-cred-b6275f9d6650\"" Nov 03 20:41:05.309462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.292104 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d8d0fa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d8d0fa\"" Nov 03 20:41:05.318120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.311096 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials\") pod \"c9555cb9-159a-4784-b6f9-7778ba070ec5\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " Nov 03 20:41:05.318120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.311386 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume\") pod \"c9555cb9-159a-4784-b6f9-7778ba070ec5\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " Nov 03 20:41:05.318120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.311443 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume\") pod \"c9555cb9-159a-4784-b6f9-7778ba070ec5\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " Nov 03 20:41:05.318120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.311489 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq\") pod \"c9555cb9-159a-4784-b6f9-7778ba070ec5\" (UID: \"c9555cb9-159a-4784-b6f9-7778ba070ec5\") " Nov 03 20:41:05.318120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.315767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume" pod "c9555cb9-159a-4784-b6f9-7778ba070ec5" (UID: "c9555cb9-159a-4784-b6f9-7778ba070ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:05.330693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.318906 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume" pod "c9555cb9-159a-4784-b6f9-7778ba070ec5" (UID: "c9555cb9-159a-4784-b6f9-7778ba070ec5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:05.364848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.364656 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v"] Nov 03 20:41:05.431592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.428034 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn4pq\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.431592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.428184 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.431592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.428438 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.431592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.428472 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.431592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.428680 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:05.431592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.428700 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9555cb9-159a-4784-b6f9-7778ba070ec5-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:05.538545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.533860 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq" pod "c9555cb9-159a-4784-b6f9-7778ba070ec5" (UID: "c9555cb9-159a-4784-b6f9-7778ba070ec5"). InnerVolumeSpecName "kube-api-access-8q5cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:05.538545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.535253 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn4pq\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.538545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.535405 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.588406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.588365 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.627536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.625035 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:05.627536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.625094 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:05.627536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.625730 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d8d0fa\"/\"openshift-service-ca.crt\"" Nov 03 20:41:05.627536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.625833 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d8d0fa\"/\"kube-root-ca.crt\"" Nov 03 20:41:05.660808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.659966 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:05.660808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.660029 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:05.703857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.693936 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials" pod "c9555cb9-159a-4784-b6f9-7778ba070ec5" (UID: "c9555cb9-159a-4784-b6f9-7778ba070ec5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:05.715544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.705258 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.715544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.705315 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.715544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.705653 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q5cq\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-kube-api-access-8q5cq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:05.715544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.705677 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9555cb9-159a-4784-b6f9-7778ba070ec5-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:05.727534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.720445 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.846307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.846238 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.888690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.888653 5815 generic.go:356] "Generic (PLEG): container finished" podID="c9555cb9-159a-4784-b6f9-7778ba070ec5" containerID="b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe" exitCode=0 Nov 03 20:41:05.894740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.890414 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" Nov 03 20:41:05.895353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.894544 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" event={"ID":"c9555cb9-159a-4784-b6f9-7778ba070ec5","Type":"ContainerDied","Data":"b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe"} Nov 03 20:41:05.895579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.895558 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d" event={"ID":"c9555cb9-159a-4784-b6f9-7778ba070ec5","Type":"ContainerDied","Data":"6c5308b3751bd4b77a3a170b82ecf0f4b59d3bd815f6e4e5906ad70a0a1b712a"} Nov 03 20:41:05.896222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.896112 5815 scope.go:122] "RemoveContainer" containerID="b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe" Nov 03 20:41:05.971684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.956402 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn4pq\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq\") pod \"payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:05.996529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:05.992830 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:06.301873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.292922 5815 scope.go:122] "RemoveContainer" containerID="b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe" Nov 03 20:41:06.351071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.350539 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j"] Nov 03 20:41:06.357438 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:06.355287 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe\": container with ID starting with b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe not found: ID does not exist" containerID="b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe" Nov 03 20:41:06.361409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.361349 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe"} err="failed to get container status \"b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe\": rpc error: code = NotFound desc = could not find container \"b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe\": container with ID starting with b2f06b7129adc2c6a5c0add825b85846c6b3e7758665f3b6ef5185d7b58291fe not found: ID does not exist" Nov 03 20:41:06.471228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.471186 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:06.607105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.607015 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:06.607399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.607363 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:06.682624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.682588 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:06.682941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.682904 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:06.752837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:06.751746 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.052058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.051995 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" podUID="38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" containerName="payload-processing" containerID="cri-o://a7ba373c523bc5c0f647f72328892207d93c7e3dcb65473671f9a8850deb9226" gracePeriod=25 Nov 03 20:41:07.293691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.293210 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:07.310571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.310475 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.311195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.311171 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822gn\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.311663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.311635 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.319572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.315428 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.533010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.527877 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j"] Nov 03 20:41:07.533010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.527950 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf"] Nov 03 20:41:07.533431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.533396 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:07.533672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.533641 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:07.564670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.551490 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.564956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.564931 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.565182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.565159 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.565444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.565422 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822gn\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.566063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.566039 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.567267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.567239 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.567886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.567862 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.569658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.569633 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:07.603117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.602811 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf"] Nov 03 20:41:07.613598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.613561 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d"] Nov 03 20:41:07.626534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.620669 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-6997xsj8d"] Nov 03 20:41:07.629954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.629927 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9"] Nov 03 20:41:07.673771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.641908 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.673771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.657120 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:07.673771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.657186 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:07.673771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.658414 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-822gn\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.840438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.791178 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.840711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.831807 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:07.841189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.841145 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxxv\" (UniqueName: \"kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.841439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.841417 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.855880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.854628 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.951559 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca\") pod \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.951961 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca" pod "3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" (UID: "3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.952038 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca\") pod \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.952769 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks\" (UniqueName: \"kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks\") pod \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\" (UID: \"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64\") " Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.953281 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxxv\" (UniqueName: \"kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.953343 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.953596 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.953800 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:07.956558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.954713 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.958480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.958446 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca" pod "3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" (UID: "3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:07.959387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.959359 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:07.959802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:07.959776 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks" pod "3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" (UID: "3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64"). InnerVolumeSpecName "kube-api-access-2n6ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:08.008541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.008218 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxxv\" (UniqueName: \"kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv\") pod \"maas-api-revoke-sub-915549f5-jpncf\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:08.056099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.055851 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n6ks\" (UniqueName: \"kubernetes.io/projected/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-kube-api-access-2n6ks\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:08.056500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.056439 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:08.177641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.172690 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:08.559474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.557825 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:08.559474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.557892 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:08.689583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.676581 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:08.689583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.676635 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:08.692742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.690424 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9555cb9-159a-4784-b6f9-7778ba070ec5" path="/var/lib/kubelet/pods/c9555cb9-159a-4784-b6f9-7778ba070ec5/volumes" Nov 03 20:41:08.759599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.759443 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v"] Nov 03 20:41:08.935126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.887328 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" Nov 03 20:41:08.935126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.888000 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk" event={"ID":"3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64","Type":"ContainerDied","Data":"63352fb38c1818b475b9ba895ab25bb92836e57cc92d9f5a12fbc5db1cb31277"} Nov 03 20:41:08.935126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:08.888049 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63352fb38c1818b475b9ba895ab25bb92836e57cc92d9f5a12fbc5db1cb31277" Nov 03 20:41:09.326872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.306299 5815 generic.go:356] "Generic (PLEG): container finished" podID="38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" containerID="a7ba373c523bc5c0f647f72328892207d93c7e3dcb65473671f9a8850deb9226" exitCode=0 Nov 03 20:41:09.326872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.306480 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" event={"ID":"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b","Type":"ContainerDied","Data":"a7ba373c523bc5c0f647f72328892207d93c7e3dcb65473671f9a8850deb9226"} Nov 03 20:41:09.326872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.306706 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:09.406684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.349914 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-7jhqb"] Nov 03 20:41:09.406684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.364454 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" containerName="revoke-keys" Nov 03 20:41:09.406684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.364478 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" containerName="revoke-keys" Nov 03 20:41:09.406684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.364888 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" containerName="revoke-keys" Nov 03 20:41:09.496832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.496801 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-7jhqb"] Nov 03 20:41:09.522362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.513963 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.515875 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.536499 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.552472 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8fdb8e61542\"" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.552700 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.553799 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8dm\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.553834 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.553952 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.614283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.554095 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.702908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.694271 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.702908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.694422 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.765625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.765548 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:09.855672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:09.846476 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:10.176479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.176435 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:10.177430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.177351 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8dm\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:10.181534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.177992 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:10.208540 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.183074 5815 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8fdb8e61542: secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:10.208804 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.208781 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555d99b548-7jhqb: secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:10.277555 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.254755 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials podName:4a6e89d6-eb57-4e49-b0d7-491143369e89 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:10.75466249 +0000 UTC m=+2380.285008167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials") pod "payload-processing-external-model-555d99b548-7jhqb" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89") : secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:10.343994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.312425 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:10.369735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.369687 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:10.402662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.402616 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 20:41:10.467439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.410325 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\") pod \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " Nov 03 20:41:10.468051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.468022 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials\") pod \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " Nov 03 20:41:10.468383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.468364 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume\") pod \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " Nov 03 20:41:10.468620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.468598 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume\") pod \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " Nov 03 20:41:10.475529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.412311 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:41:10.507011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.506753 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials" pod "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" (UID: "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:10.507011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.506763 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:10.507538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.507362 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume" pod "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" (UID: "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:10.510412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.507910 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume" pod "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" (UID: "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:10.510412 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.508369 5815 projected.go:196] Error preparing data for projected volume kube-api-access-9f8dm for pod models-as-a-service/payload-processing-external-model-555d99b548-7jhqb: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Nov 03 20:41:10.510412 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.508439 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm podName:4a6e89d6-eb57-4e49-b0d7-491143369e89 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:11.008418957 +0000 UTC m=+2380.538764625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9f8dm" (UniqueName: "kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm") pod "payload-processing-external-model-555d99b548-7jhqb" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Nov 03 20:41:10.512129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.511838 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:10.519053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.519016 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:41:10.521115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.521086 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:10.525048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.525023 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:10.525233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.525211 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:10.666611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.651800 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2" pod "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" (UID: "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b"). InnerVolumeSpecName "kube-api-access-f5br2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:10.758073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.758042 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\") pod \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\" (UID: \"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b\") " Nov 03 20:41:10.760289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.760257 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:10.760796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.760764 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:10.760935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.760921 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:10.761032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.761019 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:10.761545 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:10.759223 5815 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b/volumes/kubernetes.io~projected/kube-api-access-f5br2 Nov 03 20:41:10.761727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:10.761707 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2" pod "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" (UID: "38f17202-f7cc-4935-b1bb-c7fb8c52ba0b"). InnerVolumeSpecName "kube-api-access-f5br2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:10.762233 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.762215 5815 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8fdb8e61542: secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:10.762365 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.762354 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555d99b548-7jhqb: secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:10.762556 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:10.762543 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials podName:4a6e89d6-eb57-4e49-b0d7-491143369e89 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:11.762485931 +0000 UTC m=+2381.292831595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials") pod "payload-processing-external-model-555d99b548-7jhqb" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89") : secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:11.011289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.011205 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5br2\" (UniqueName: \"kubernetes.io/projected/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b-kube-api-access-f5br2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:11.101577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.064054 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" Nov 03 20:41:11.117525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.117470 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" event={"ID":"b0989e9a-50f9-4762-8f6f-8119decc007d","Type":"ContainerStarted","Data":"d54d4e442b53102052213105fc426fb5f075ecd19ff767031f9fcc9254c86bc6"} Nov 03 20:41:11.117817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.117795 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-7jhqb"] Nov 03 20:41:11.162530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.136620 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5" event={"ID":"38f17202-f7cc-4935-b1bb-c7fb8c52ba0b","Type":"ContainerDied","Data":"fed0cedcad88e1d8d96b453e4788a921194a3272dc508b7c4490cc1393d97321"} Nov 03 20:41:11.262820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.220796 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:11.263114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.263084 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:11.256988 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9f8dm provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" podUID="4a6e89d6-eb57-4e49-b0d7-491143369e89" Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.240223 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8dm\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:11.258829 5815 projected.go:196] Error preparing data for projected volume kube-api-access-9f8dm for pod models-as-a-service/payload-processing-external-model-555d99b548-7jhqb: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:11.331185 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm podName:4a6e89d6-eb57-4e49-b0d7-491143369e89 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:12.331141535 +0000 UTC m=+2381.861487188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9f8dm" (UniqueName: "kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm") pod "payload-processing-external-model-555d99b548-7jhqb" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.286382 5815 scope.go:122] "RemoveContainer" containerID="a7ba373c523bc5c0f647f72328892207d93c7e3dcb65473671f9a8850deb9226" Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.286017 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:41:11.332085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.331461 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j"] Nov 03 20:41:11.452563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.449339 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5"] Nov 03 20:41:11.452563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.449390 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-699dt8bz5"] Nov 03 20:41:11.518712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.492164 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf"] Nov 03 20:41:11.554532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.512340 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:11.554532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.540745 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:11.653933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.653902 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:11.654090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.653950 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:11.685659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.685561 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" event={"ID":"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7","Type":"ContainerStarted","Data":"07c6964c5db3c2f5516c4b83aa1dcd45222191cd560f70a25e38a488accb3589"} Nov 03 20:41:11.702779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.702689 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" event={"ID":"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5","Type":"ContainerStarted","Data":"3c09a05ca49735fa3c063ff5da472b21dd2781bbc2d4951dd5ed7eebe8b2d209"} Nov 03 20:41:11.713606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.705305 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" event={"ID":"b0989e9a-50f9-4762-8f6f-8119decc007d","Type":"ContainerStarted","Data":"e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5"} Nov 03 20:41:11.713606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.706632 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:11.752299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.751690 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:11.784320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.784229 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:11.785528 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:11.785481 5815 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b8fdb8e61542: secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:11.798460 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:11.798434 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-555d99b548-7jhqb: secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:11.798797 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:11.798780 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials podName:4a6e89d6-eb57-4e49-b0d7-491143369e89 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:13.798750935 +0000 UTC m=+2383.329096590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials") pod "payload-processing-external-model-555d99b548-7jhqb" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89") : secret "agc-cred-b8fdb8e61542" not found Nov 03 20:41:11.799244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.786360 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:11.801267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.791125 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" podStartSLOduration=6.791106673 podStartE2EDuration="6.791106673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:11.762929052 +0000 UTC m=+2381.293274716" watchObservedRunningTime="2026-11-03 20:41:11.791106673 +0000 UTC m=+2381.321452394" Nov 03 20:41:11.909851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.909798 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume\") pod \"4a6e89d6-eb57-4e49-b0d7-491143369e89\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " Nov 03 20:41:11.910638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.910608 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume\") pod \"4a6e89d6-eb57-4e49-b0d7-491143369e89\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " Nov 03 20:41:11.912093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.912062 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume" pod "4a6e89d6-eb57-4e49-b0d7-491143369e89" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:11.920561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:11.912823 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume" pod "4a6e89d6-eb57-4e49-b0d7-491143369e89" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:12.066609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.059972 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:12.066609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.060008 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a6e89d6-eb57-4e49-b0d7-491143369e89-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:12.389599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.388949 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8dm\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm\") pod \"payload-processing-external-model-555d99b548-7jhqb\" (UID: \"4a6e89d6-eb57-4e49-b0d7-491143369e89\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:12.413529 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:12.403643 5815 projected.go:196] Error preparing data for projected volume kube-api-access-9f8dm for pod models-as-a-service/payload-processing-external-model-555d99b548-7jhqb: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Nov 03 20:41:12.413529 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:12.403817 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm podName:4a6e89d6-eb57-4e49-b0d7-491143369e89 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:14.403794923 +0000 UTC m=+2383.934140595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9f8dm" (UniqueName: "kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm") pod "payload-processing-external-model-555d99b548-7jhqb" (UID: "4a6e89d6-eb57-4e49-b0d7-491143369e89") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Nov 03 20:41:12.531597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.499903 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:12.531597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.499980 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:12.648223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.648112 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" podUID="54a49df6-e295-4fd5-84f5-9f6cbff4b29e" containerName="payload-processing" containerID="cri-o://b0fd6552d4acdc1bf0a74725d29d87dc0c9ebbfc58bf39b7a218a29e5ab802eb" gracePeriod=25 Nov 03 20:41:12.686094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.686056 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:12.687325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.686795 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:12.858538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.856339 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" path="/var/lib/kubelet/pods/38f17202-f7cc-4935-b1bb-c7fb8c52ba0b/volumes" Nov 03 20:41:12.858538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.857148 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r"] Nov 03 20:41:12.862167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.862142 5815 cpu_manager.go:414] "containerMap: removing container" podUID="38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" containerName="payload-processing" Nov 03 20:41:12.862167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.862167 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" containerName="payload-processing" Nov 03 20:41:12.862619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.862499 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f17202-f7cc-4935-b1bb-c7fb8c52ba0b" containerName="payload-processing" Nov 03 20:41:12.900573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.888074 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r"] Nov 03 20:41:12.900573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.888163 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:12.900573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.900444 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-e2e--0dce6837-dockercfg-c69fq\"" Nov 03 20:41:12.900857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.900579 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-plugins-e2e-extmod-a-37aa95\"" Nov 03 20:41:12.900857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.900647 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-b599b816968c\"" Nov 03 20:41:12.900857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.900706 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-5797404fedc5\"" Nov 03 20:41:12.900857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.900788 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"routing-overlay\"" Nov 03 20:41:12.928531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.925528 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"kube-root-ca.crt\"" Nov 03 20:41:12.928531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.925643 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"openshift-service-ca.crt\"" Nov 03 20:41:12.928531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.928060 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" event={"ID":"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7","Type":"ContainerStarted","Data":"f2bb6fc98396467f174e21392960d743fcafd3962aa02316102efdfe1833e941"} Nov 03 20:41:12.940343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.940316 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" event={"ID":"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5","Type":"ContainerStarted","Data":"7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340"} Nov 03 20:41:12.940523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.940396 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-7jhqb" Nov 03 20:41:12.961355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:12.961310 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" podStartSLOduration=7.961268802 podStartE2EDuration="7.961268802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:12.959865596 +0000 UTC m=+2382.490211264" watchObservedRunningTime="2026-11-03 20:41:12.961268802 +0000 UTC m=+2382.491614488" Nov 03 20:41:13.047177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.047118 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" podStartSLOduration=8.047096647 podStartE2EDuration="8.047096647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:12.996673589 +0000 UTC m=+2382.527019263" watchObservedRunningTime="2026-11-03 20:41:13.047096647 +0000 UTC m=+2382.577442322" Nov 03 20:41:13.054965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.054887 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.055183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.055029 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkm6h\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.055183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.055121 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.055183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.055176 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.145375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.144603 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-7jhqb"] Nov 03 20:41:13.158743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.158661 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkm6h\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.159108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.159083 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.159297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.159278 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.159779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.159753 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.165537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.160887 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.187533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.168847 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.187759 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.168950 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-b599b816968c: secret "agc-cred-b599b816968c" not found Nov 03 20:41:13.187879 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.187863 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r: secret "agc-cred-b599b816968c" not found Nov 03 20:41:13.188031 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.188019 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials podName:8b33045f-3d3d-45dc-8757-6702ce6e2911 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:13.687993138 +0000 UTC m=+2383.218338800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" (UID: "8b33045f-3d3d-45dc-8757-6702ce6e2911") : secret "agc-cred-b599b816968c" not found Nov 03 20:41:13.216941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.200161 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-7jhqb"] Nov 03 20:41:13.295532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.287875 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r"] Nov 03 20:41:13.295532 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.288492 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xkm6h provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" podUID="8b33045f-3d3d-45dc-8757-6702ce6e2911" Nov 03 20:41:13.361676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.346911 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:13.361676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.346949 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8dm\" (UniqueName: \"kubernetes.io/projected/4a6e89d6-eb57-4e49-b0d7-491143369e89-kube-api-access-9f8dm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:13.361676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.349313 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkm6h\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.526647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.444617 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8"] Nov 03 20:41:13.596533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.596376 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:13.596533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.596436 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:13.709537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.685760 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:13.709537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.685825 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:13.709537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.695398 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:13.709537 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.695886 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-b599b816968c: secret "agc-cred-b599b816968c" not found Nov 03 20:41:13.709537 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.695907 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r: secret "agc-cred-b599b816968c" not found Nov 03 20:41:13.709537 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:13.695965 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials podName:8b33045f-3d3d-45dc-8757-6702ce6e2911 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:14.695947269 +0000 UTC m=+2384.226292922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" (UID: "8b33045f-3d3d-45dc-8757-6702ce6e2911") : secret "agc-cred-b599b816968c" not found Nov 03 20:41:13.733690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.723625 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:13.862596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.862564 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:13.877578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.877545 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cz6\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:13.877923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.877889 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:13.878326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.878302 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.007533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:13.982070 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.041000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.040935 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cz6\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.041773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.041554 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.046486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.046463 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.047008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.046387 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.047130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.037928 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.060951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.060910 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.061484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.061460 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cz6\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.142582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.119791 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8"] Nov 03 20:41:14.155643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.155429 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b"] Nov 03 20:41:14.191367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.191110 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:14.216821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.216241 5815 generic.go:356] "Generic (PLEG): container finished" podID="d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" containerID="f2bb6fc98396467f174e21392960d743fcafd3962aa02316102efdfe1833e941" exitCode=0 Nov 03 20:41:14.223000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.220294 5815 generic.go:356] "Generic (PLEG): container finished" podID="54a49df6-e295-4fd5-84f5-9f6cbff4b29e" containerID="b0fd6552d4acdc1bf0a74725d29d87dc0c9ebbfc58bf39b7a218a29e5ab802eb" exitCode=0 Nov 03 20:41:14.223000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.220596 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:14.241579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.239474 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.241579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.239380 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b"] Nov 03 20:41:14.241579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.239602 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" event={"ID":"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7","Type":"ContainerDied","Data":"f2bb6fc98396467f174e21392960d743fcafd3962aa02316102efdfe1833e941"} Nov 03 20:41:14.241579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.239642 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:14.241579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.239664 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:41:14.241579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.239691 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" event={"ID":"54a49df6-e295-4fd5-84f5-9f6cbff4b29e","Type":"ContainerDied","Data":"b0fd6552d4acdc1bf0a74725d29d87dc0c9ebbfc58bf39b7a218a29e5ab802eb"} Nov 03 20:41:14.253547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.253232 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:41:14.258166 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:14.258118 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:41:14.284587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.284559 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:14.361438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.361402 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials\") pod \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " Nov 03 20:41:14.361712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.361694 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume\") pod \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " Nov 03 20:41:14.362587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.362566 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs\") pod \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " Nov 03 20:41:14.362806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.362791 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume\") pod \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\" (UID: \"54a49df6-e295-4fd5-84f5-9f6cbff4b29e\") " Nov 03 20:41:14.363478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.363454 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.364108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.364087 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.364424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.364406 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qcz\" (UniqueName: \"kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.372580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.364861 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials" pod "54a49df6-e295-4fd5-84f5-9f6cbff4b29e" (UID: "54a49df6-e295-4fd5-84f5-9f6cbff4b29e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:14.372580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.365425 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume" pod "54a49df6-e295-4fd5-84f5-9f6cbff4b29e" (UID: "54a49df6-e295-4fd5-84f5-9f6cbff4b29e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:14.375062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.375039 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume" pod "54a49df6-e295-4fd5-84f5-9f6cbff4b29e" (UID: "54a49df6-e295-4fd5-84f5-9f6cbff4b29e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:14.382798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.382767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs" pod "54a49df6-e295-4fd5-84f5-9f6cbff4b29e" (UID: "54a49df6-e295-4fd5-84f5-9f6cbff4b29e"). InnerVolumeSpecName "kube-api-access-5f9vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:14.474245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.474214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume\") pod \"8b33045f-3d3d-45dc-8757-6702ce6e2911\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " Nov 03 20:41:14.474897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.474879 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h\") pod \"8b33045f-3d3d-45dc-8757-6702ce6e2911\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " Nov 03 20:41:14.475237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.474612 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume" pod "8b33045f-3d3d-45dc-8757-6702ce6e2911" (UID: "8b33045f-3d3d-45dc-8757-6702ce6e2911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:14.475797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.475746 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume\") pod \"8b33045f-3d3d-45dc-8757-6702ce6e2911\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " Nov 03 20:41:14.476201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.476169 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume" pod "8b33045f-3d3d-45dc-8757-6702ce6e2911" (UID: "8b33045f-3d3d-45dc-8757-6702ce6e2911"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:14.477890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.476725 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.478201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.477474 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.478471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.478455 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qcz\" (UniqueName: \"kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.478849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.478498 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h" pod "8b33045f-3d3d-45dc-8757-6702ce6e2911" (UID: "8b33045f-3d3d-45dc-8757-6702ce6e2911"). InnerVolumeSpecName "kube-api-access-xkm6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:14.479004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.478985 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.479375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.479356 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.479527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.479494 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.479661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.479647 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.479756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.479743 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkm6h\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-kube-api-access-xkm6h\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.479848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.479836 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9vs\" (UniqueName: \"kubernetes.io/projected/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-kube-api-access-5f9vs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.480011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.479997 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54a49df6-e295-4fd5-84f5-9f6cbff4b29e-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.480130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.480118 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8b33045f-3d3d-45dc-8757-6702ce6e2911-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:14.483553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.483526 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.492177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.492154 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qcz\" (UniqueName: \"kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz\") pod \"maas-api-revoke-sub-acc87b8c-h9f7b\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.497257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.497235 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:14.497449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.497426 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:14.575132 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:14.575094 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45364c34_1fee_4841_9d6f_15818e90d572.slice/crio-6e90aa063563ee56f486a34b28ef3c094f9f06947affe34d4684939fb91d456e WatchSource:0}: Error finding container 6e90aa063563ee56f486a34b28ef3c094f9f06947affe34d4684939fb91d456e: Status 404 returned error can't find the container with id 6e90aa063563ee56f486a34b28ef3c094f9f06947affe34d4684939fb91d456e Nov 03 20:41:14.576944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.576918 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8"] Nov 03 20:41:14.595376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.595351 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6e89d6-eb57-4e49-b0d7-491143369e89" path="/var/lib/kubelet/pods/4a6e89d6-eb57-4e49-b0d7-491143369e89/volumes" Nov 03 20:41:14.599274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.599040 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:14.647899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.647444 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:14.647899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.647523 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:14.799536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:14.798558 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r\" (UID: \"8b33045f-3d3d-45dc-8757-6702ce6e2911\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:14.800951 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:14.800541 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-b599b816968c: secret "agc-cred-b599b816968c" not found Nov 03 20:41:14.800951 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:14.800565 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r: secret "agc-cred-b599b816968c" not found Nov 03 20:41:14.800951 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:14.800625 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials podName:8b33045f-3d3d-45dc-8757-6702ce6e2911 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:16.800604538 +0000 UTC m=+2386.330950201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" (UID: "8b33045f-3d3d-45dc-8757-6702ce6e2911") : secret "agc-cred-b599b816968c" not found Nov 03 20:41:15.004283 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:15.004235 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95422424_c192_400e_8c4c_40b48c4a9d25.slice/crio-cfcaee17013974e1a5d02fdacd29556546fdd11a76a0558793316a02c0d8f637 WatchSource:0}: Error finding container cfcaee17013974e1a5d02fdacd29556546fdd11a76a0558793316a02c0d8f637: Status 404 returned error can't find the container with id cfcaee17013974e1a5d02fdacd29556546fdd11a76a0558793316a02c0d8f637 Nov 03 20:41:15.004548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.004482 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b"] Nov 03 20:41:15.234926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.234886 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" event={"ID":"95422424-c192-400e-8c4c-40b48c4a9d25","Type":"ContainerStarted","Data":"cfcaee17013974e1a5d02fdacd29556546fdd11a76a0558793316a02c0d8f637"} Nov 03 20:41:15.237116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.237084 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" event={"ID":"45364c34-1fee-4841-9d6f-15818e90d572","Type":"ContainerStarted","Data":"6023ddf922b6193f189db5b6960d4a8dc64e2dcd5272c79c11b656a0c7fd7dc1"} Nov 03 20:41:15.237290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.237143 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" event={"ID":"45364c34-1fee-4841-9d6f-15818e90d572","Type":"ContainerStarted","Data":"6e90aa063563ee56f486a34b28ef3c094f9f06947affe34d4684939fb91d456e"} Nov 03 20:41:15.237290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.237213 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:15.238796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.238778 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" Nov 03 20:41:15.238955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.238820 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9" event={"ID":"54a49df6-e295-4fd5-84f5-9f6cbff4b29e","Type":"ContainerDied","Data":"26146c74787654a3c495ce52eddb8cf1fdf8ff9abaccc999b25c4d9acb0a6ecd"} Nov 03 20:41:15.238955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.238868 5815 scope.go:122] "RemoveContainer" containerID="b0fd6552d4acdc1bf0a74725d29d87dc0c9ebbfc58bf39b7a218a29e5ab802eb" Nov 03 20:41:15.239281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.239254 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r" Nov 03 20:41:15.276429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.276380 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" podStartSLOduration=2.276364607 podStartE2EDuration="2.276364607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:15.271675381 +0000 UTC m=+2384.802021055" watchObservedRunningTime="2026-11-03 20:41:15.276364607 +0000 UTC m=+2384.806710278" Nov 03 20:41:15.335848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.335787 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r"] Nov 03 20:41:15.345103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.345056 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7dd4hw68r"] Nov 03 20:41:15.355415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.355357 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9"] Nov 03 20:41:15.363809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.363782 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6484f5959b-6ftn9"] Nov 03 20:41:15.500539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.489378 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8b33045f-3d3d-45dc-8757-6702ce6e2911-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:15.529023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.528942 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:15.529277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.529252 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:15.699542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.677844 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:15.699542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.677922 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:15.917536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:15.916688 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:16.016975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.016940 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca\") pod \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " Nov 03 20:41:16.017499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.017461 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv\" (UniqueName: \"kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv\") pod \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " Nov 03 20:41:16.018457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.018434 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca\") pod \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\" (UID: \"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7\") " Nov 03 20:41:16.020114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.020074 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca" pod "d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" (UID: "d0b2dc00-9f61-4bfe-8775-86cad0d4fee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:16.023200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.023167 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca" pod "d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" (UID: "d0b2dc00-9f61-4bfe-8775-86cad0d4fee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:16.040548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.037763 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv" pod "d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" (UID: "d0b2dc00-9f61-4bfe-8775-86cad0d4fee7"). InnerVolumeSpecName "kube-api-access-hsxxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:16.172931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.172840 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.201528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.201477 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.208607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.208574 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxxv\" (UniqueName: \"kubernetes.io/projected/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7-kube-api-access-hsxxv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:16.553529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.549120 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:16.688807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.688770 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:16.689071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.689042 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:16.708531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.692872 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:16.708864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.708829 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:16.848699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:16.800730 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" Nov 03 20:41:17.047560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.047519 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:17.089671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.089621 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:41:17.165812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.145764 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a49df6-e295-4fd5-84f5-9f6cbff4b29e" path="/var/lib/kubelet/pods/54a49df6-e295-4fd5-84f5-9f6cbff4b29e/volumes" Nov 03 20:41:17.175554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.162375 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" podStartSLOduration=4.162352611 podStartE2EDuration="4.162352611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:16.940032222 +0000 UTC m=+2386.470377895" watchObservedRunningTime="2026-11-03 20:41:17.162352611 +0000 UTC m=+2386.692698284" Nov 03 20:41:17.179456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.179412 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b33045f-3d3d-45dc-8757-6702ce6e2911" path="/var/lib/kubelet/pods/8b33045f-3d3d-45dc-8757-6702ce6e2911/volumes" Nov 03 20:41:17.201269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.201228 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf" event={"ID":"d0b2dc00-9f61-4bfe-8775-86cad0d4fee7","Type":"ContainerDied","Data":"07c6964c5db3c2f5516c4b83aa1dcd45222191cd560f70a25e38a488accb3589"} Nov 03 20:41:17.201556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.201532 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c6964c5db3c2f5516c4b83aa1dcd45222191cd560f70a25e38a488accb3589" Nov 03 20:41:17.201693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.201677 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-t5mdw"] Nov 03 20:41:17.222538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.212610 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" containerName="revoke-keys" Nov 03 20:41:17.251673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.251638 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" containerName="revoke-keys" Nov 03 20:41:17.251978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.251964 5815 cpu_manager.go:414] "containerMap: removing container" podUID="54a49df6-e295-4fd5-84f5-9f6cbff4b29e" containerName="payload-processing" Nov 03 20:41:17.252076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.252065 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a49df6-e295-4fd5-84f5-9f6cbff4b29e" containerName="payload-processing" Nov 03 20:41:17.269912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.269878 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" containerName="revoke-keys" Nov 03 20:41:17.270185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.270170 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a49df6-e295-4fd5-84f5-9f6cbff4b29e" containerName="payload-processing" Nov 03 20:41:17.271763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.238632 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:17.305702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.305671 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:17.306145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.306121 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" event={"ID":"95422424-c192-400e-8c4c-40b48c4a9d25","Type":"ContainerStarted","Data":"630685e305f2051237082e07178df400d3c584c8715a4cc63364a060faafae6f"} Nov 03 20:41:17.333972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.333931 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-t5mdw"] Nov 03 20:41:17.334238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.305979 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.351314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.351287 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:41:17.364563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.352181 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b8fdb8e61542\"" Nov 03 20:41:17.364809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.353989 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-fldbr\"" Nov 03 20:41:17.364917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.354101 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:41:17.365002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.354186 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:41:17.365083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.354280 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:41:17.439727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.439634 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.440131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.440111 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.440450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.440433 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.471546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.471205 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64nph\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.472077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.472041 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:17.516749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.512128 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:17.516749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.512361 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:41:17.516749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.513683 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:17.516749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.513730 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:17.602541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.602393 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.615538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.603021 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.615538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.606661 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64nph\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.615538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.606831 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.630534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.628601 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.631227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.631197 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.651688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.651651 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:17.651936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.651907 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:17.685501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.685466 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64nph\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.691727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.691652 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials\") pod \"payload-processing-external-model-555d99b548-t5mdw\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.766073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.764639 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:41:17.906416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.906320 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:17.950029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:17.949934 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:18.110538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.108928 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:18.113209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.113179 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:41:18.117071 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.117028 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9"] Nov 03 20:41:18.135134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.134750 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:41:18.139072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.139049 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:41:18.140015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.139996 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.165882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.164863 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"kube-root-ca.crt\"" Nov 03 20:41:18.166422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.166233 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"openshift-service-ca.crt\"" Nov 03 20:41:18.166422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.166307 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-plugins-e2e-extmod-a-db5198\"" Nov 03 20:41:18.166422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.166385 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"routing-overlay\"" Nov 03 20:41:18.166825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.166793 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-e2e--e92303d8-dockercfg-jf5p4\"" Nov 03 20:41:18.195148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.195059 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9"] Nov 03 20:41:18.211131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.210457 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:41:18.217413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.211747 5815 generic.go:356] "Generic (PLEG): container finished" podID="95422424-c192-400e-8c4c-40b48c4a9d25" containerID="630685e305f2051237082e07178df400d3c584c8715a4cc63364a060faafae6f" exitCode=0 Nov 03 20:41:18.217413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.211929 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" event={"ID":"95422424-c192-400e-8c4c-40b48c4a9d25","Type":"ContainerDied","Data":"630685e305f2051237082e07178df400d3c584c8715a4cc63364a060faafae6f"} Nov 03 20:41:18.234105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.234077 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wlj\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.235126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.235101 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.235951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.235930 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.236165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.236133 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.244927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.244775 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"agc-cred-79526f366f5b\"" Nov 03 20:41:18.353921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.353864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.354535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.354497 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.354744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.354726 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.354985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.354969 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wlj\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.359279 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.356397 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.369836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.369813 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.412995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.412969 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:41:18.420109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.420075 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.425815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.425617 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wlj\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.516529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.511873 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:18.516529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.511934 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:18.551538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.551486 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:18.657555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.651224 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:18.657555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:18.651488 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:19.069032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.048727 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2"] Nov 03 20:41:19.152578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.126170 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.187531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.140559 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2"] Nov 03 20:41:19.300438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.300413 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"agc-cred-0c2c8ae114be\"" Nov 03 20:41:19.301859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.301839 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"payload-processing-external-model-e2e--4ce1836f-dockercfg-hzzv9\"" Nov 03 20:41:19.302231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.302214 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"openshift-service-ca.crt\"" Nov 03 20:41:19.302438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.302423 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"kube-root-ca.crt\"" Nov 03 20:41:19.302570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.302489 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"routing-overlay\"" Nov 03 20:41:19.302655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.302591 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"payload-processing-external-model-plugins-e2e-extmod-b-421604\"" Nov 03 20:41:19.310767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.310742 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-t5mdw"] Nov 03 20:41:19.438989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.411454 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.438989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.438882 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.439258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.439155 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hghvf\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.439258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.439248 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.515149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.503552 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body= Nov 03 20:41:19.515149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.503607 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" Nov 03 20:41:19.548536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.548028 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hghvf\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.548536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.548123 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.548536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.548327 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.548536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.548379 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.552567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.549340 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.552567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.550576 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.554364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.554340 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.569263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.569240 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hghvf\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.647362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.647326 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Nov 03 20:41:19.647575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.647407 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Nov 03 20:41:19.651641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.651616 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:19.752231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.752198 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9"] Nov 03 20:41:19.794548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.792599 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:19.853054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.853025 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca\") pod \"95422424-c192-400e-8c4c-40b48c4a9d25\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " Nov 03 20:41:19.853198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.853165 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca\") pod \"95422424-c192-400e-8c4c-40b48c4a9d25\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " Nov 03 20:41:19.854401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.854379 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz\" (UniqueName: \"kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz\") pod \"95422424-c192-400e-8c4c-40b48c4a9d25\" (UID: \"95422424-c192-400e-8c4c-40b48c4a9d25\") " Nov 03 20:41:19.856018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.855995 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca" pod "95422424-c192-400e-8c4c-40b48c4a9d25" (UID: "95422424-c192-400e-8c4c-40b48c4a9d25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:19.891924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.869586 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca" pod "95422424-c192-400e-8c4c-40b48c4a9d25" (UID: "95422424-c192-400e-8c4c-40b48c4a9d25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:19.891924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.889076 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz" pod "95422424-c192-400e-8c4c-40b48c4a9d25" (UID: "95422424-c192-400e-8c4c-40b48c4a9d25"). InnerVolumeSpecName "kube-api-access-49qcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:19.956709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.956579 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:19.956709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.956613 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95422424-c192-400e-8c4c-40b48c4a9d25-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:19.956709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:19.956628 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qcz\" (UniqueName: \"kubernetes.io/projected/95422424-c192-400e-8c4c-40b48c4a9d25-kube-api-access-49qcz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:20.053395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.053334 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2"] Nov 03 20:41:20.341345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.341312 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" event={"ID":"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f","Type":"ContainerStarted","Data":"a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45"} Nov 03 20:41:20.341903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.341375 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" event={"ID":"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f","Type":"ContainerStarted","Data":"c6accfcfcebb96e330dae20951a4642d6fdc9b5045223196a1707c27a181c243"} Nov 03 20:41:20.341903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.341425 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:20.343678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.343650 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" event={"ID":"0dae5afa-11a2-482d-8f16-7949dd157d58","Type":"ContainerStarted","Data":"fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b"} Nov 03 20:41:20.343834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.343699 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" event={"ID":"0dae5afa-11a2-482d-8f16-7949dd157d58","Type":"ContainerStarted","Data":"b0ce5b19f6c86123a5a01c95f50076b74edcd09c33a94c78274a6772edaff577"} Nov 03 20:41:20.344128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.344108 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:20.345840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.345804 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" event={"ID":"7ba63e1d-67bc-40c3-ba3d-fad7fa505134","Type":"ContainerStarted","Data":"41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55"} Nov 03 20:41:20.345978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.345850 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" event={"ID":"7ba63e1d-67bc-40c3-ba3d-fad7fa505134","Type":"ContainerStarted","Data":"c56c29a3edc1af407ed7c428b63cd2d130ce3717b01d779ebf66486756461e59"} Nov 03 20:41:20.345978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.345879 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:20.348614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.348579 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" event={"ID":"95422424-c192-400e-8c4c-40b48c4a9d25","Type":"ContainerDied","Data":"cfcaee17013974e1a5d02fdacd29556546fdd11a76a0558793316a02c0d8f637"} Nov 03 20:41:20.348710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.348626 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfcaee17013974e1a5d02fdacd29556546fdd11a76a0558793316a02c0d8f637" Nov 03 20:41:20.348710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.348646 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b" Nov 03 20:41:20.378066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.370967 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" podStartSLOduration=2.370951283 podStartE2EDuration="2.370951283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:20.361140681 +0000 UTC m=+2389.891486344" watchObservedRunningTime="2026-11-03 20:41:20.370951283 +0000 UTC m=+2389.901296958" Nov 03 20:41:20.383007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.382899 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" podStartSLOduration=4.382882576 podStartE2EDuration="4.382882576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:20.380288157 +0000 UTC m=+2389.910633830" watchObservedRunningTime="2026-11-03 20:41:20.382882576 +0000 UTC m=+2389.913228249" Nov 03 20:41:20.410530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:20.403314 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" podStartSLOduration=2.4033043689999998 podStartE2EDuration="2.403304369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:20.402859014 +0000 UTC m=+2389.933204686" watchObservedRunningTime="2026-11-03 20:41:20.403304369 +0000 UTC m=+2389.933650042" Nov 03 20:41:21.496098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:21.496064 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:21.497690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:21.497484 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:41:21.497927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:21.497575 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:21.498639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:21.498616 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:41:21.679034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:21.678993 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:21.681605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:21.681567 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 20:41:22.427889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:22.427854 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-t5mdw"] Nov 03 20:41:24.236381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:24.236352 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:41:25.267236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:25.267201 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:25.570607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:25.562023 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh"] Nov 03 20:41:26.876579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:26.774581 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg"] Nov 03 20:41:26.943538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:26.903913 5815 cpu_manager.go:414] "containerMap: removing container" podUID="95422424-c192-400e-8c4c-40b48c4a9d25" containerName="revoke-keys" Nov 03 20:41:26.964140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:26.964102 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95422424-c192-400e-8c4c-40b48c4a9d25" containerName="revoke-keys" Nov 03 20:41:26.979097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:26.979054 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="95422424-c192-400e-8c4c-40b48c4a9d25" containerName="revoke-keys" Nov 03 20:41:27.065532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.064743 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg"] Nov 03 20:41:27.065532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.064863 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.097167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.096724 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:27.142641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.134943 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:41:27.142641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.135008 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:41:27.174814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.174781 5815 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" secret="" err="secret \"payload-processing-external-model-e2e--0dce6837-dockercfg-c69fq\" not found" Nov 03 20:41:27.201267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.177280 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.201770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.201744 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.201979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.201964 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bvt\" (UniqueName: \"kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.209676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.209609 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8"] Nov 03 20:41:27.420586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.415717 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.420586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.415945 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.420586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.416176 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bvt\" (UniqueName: \"kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.422955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.422924 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.501238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.458411 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" podUID="7ba63e1d-67bc-40c3-ba3d-fad7fa505134" containerName="payload-processing" containerID="cri-o://41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55" gracePeriod=25 Nov 03 20:41:27.597490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.597435 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bvt\" (UniqueName: \"kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.597808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.597498 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0b355e-8w9rg\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.636536 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:27.625600 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-5797404fedc5: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:27.655963 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:27.655933 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:27.656201 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:27.627210 5815 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:27.680416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.629562 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:27.697344 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:27.656408 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:28.156377747 +0000 UTC m=+2397.686723408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : secret "agc-cred-5797404fedc5" not found Nov 03 20:41:27.698769 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:27.698748 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:28.198714541 +0000 UTC m=+2397.729060222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:27.966357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:27.965941 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:28.014673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.014170 5815 generic.go:356] "Generic (PLEG): container finished" podID="7ba63e1d-67bc-40c3-ba3d-fad7fa505134" containerID="41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55" exitCode=0 Nov 03 20:41:28.014673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.014593 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" Nov 03 20:41:28.015146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.015107 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" event={"ID":"7ba63e1d-67bc-40c3-ba3d-fad7fa505134","Type":"ContainerDied","Data":"41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55"} Nov 03 20:41:28.015389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.015332 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-555d99b548-t5mdw" event={"ID":"7ba63e1d-67bc-40c3-ba3d-fad7fa505134","Type":"ContainerDied","Data":"c56c29a3edc1af407ed7c428b63cd2d130ce3717b01d779ebf66486756461e59"} Nov 03 20:41:28.015612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.015559 5815 scope.go:122] "RemoveContainer" containerID="41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55" Nov 03 20:41:28.036792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.036680 5815 scope.go:122] "RemoveContainer" containerID="41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55" Nov 03 20:41:28.037362 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:28.037266 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55\": container with ID starting with 41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55 not found: ID does not exist" containerID="41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55" Nov 03 20:41:28.037362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.037307 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55"} err="failed to get container status \"41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55\": rpc error: code = NotFound desc = could not find container \"41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55\": container with ID starting with 41c8f3deb17574fa7a42d5eff48e74212b6e49d1fc40e8e745542b93a6352e55 not found: ID does not exist" Nov 03 20:41:28.056990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.056944 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph\") pod \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " Nov 03 20:41:28.057243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.057220 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume\") pod \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " Nov 03 20:41:28.058326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.058271 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials\") pod \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " Nov 03 20:41:28.058602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.058581 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume\") pod \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\" (UID: \"7ba63e1d-67bc-40c3-ba3d-fad7fa505134\") " Nov 03 20:41:28.059548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.058584 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume" pod "7ba63e1d-67bc-40c3-ba3d-fad7fa505134" (UID: "7ba63e1d-67bc-40c3-ba3d-fad7fa505134"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:28.059938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.059902 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume" pod "7ba63e1d-67bc-40c3-ba3d-fad7fa505134" (UID: "7ba63e1d-67bc-40c3-ba3d-fad7fa505134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:28.061856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.061832 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph" pod "7ba63e1d-67bc-40c3-ba3d-fad7fa505134" (UID: "7ba63e1d-67bc-40c3-ba3d-fad7fa505134"). InnerVolumeSpecName "kube-api-access-64nph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:28.062060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.062035 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials" pod "7ba63e1d-67bc-40c3-ba3d-fad7fa505134" (UID: "7ba63e1d-67bc-40c3-ba3d-fad7fa505134"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:28.161205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.161172 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64nph\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-kube-api-access-64nph\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:28.161205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.161200 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:28.161205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.161210 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:28.161479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.161220 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba63e1d-67bc-40c3-ba3d-fad7fa505134-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:28.161479 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:28.161266 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-5797404fedc5: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:28.161479 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:28.161285 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:28.161479 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:28.161348 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:29.16132879 +0000 UTC m=+2398.691674443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : secret "agc-cred-5797404fedc5" not found Nov 03 20:41:28.241062 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:28.241035 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90da652a_1f80_4e01_bf1d_7ef8a5f004ea.slice/crio-c86976ba8ee5fffa5767563c0e8ae3716f6a3836d4d6f0152e9e670012ba4cf2 WatchSource:0}: Error finding container c86976ba8ee5fffa5767563c0e8ae3716f6a3836d4d6f0152e9e670012ba4cf2: Status 404 returned error can't find the container with id c86976ba8ee5fffa5767563c0e8ae3716f6a3836d4d6f0152e9e670012ba4cf2 Nov 03 20:41:28.242180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.242154 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg"] Nov 03 20:41:28.262757 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:28.262732 5815 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:28.262889 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:28.262802 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:29.26278782 +0000 UTC m=+2398.793133471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:28.439934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.439906 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-t5mdw"] Nov 03 20:41:28.446615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.446581 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-555d99b548-t5mdw"] Nov 03 20:41:28.590596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:28.590569 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba63e1d-67bc-40c3-ba3d-fad7fa505134" path="/var/lib/kubelet/pods/7ba63e1d-67bc-40c3-ba3d-fad7fa505134/volumes" Nov 03 20:41:29.065172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:29.065132 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" event={"ID":"90da652a-1f80-4e01-bf1d-7ef8a5f004ea","Type":"ContainerStarted","Data":"107a14794a1b1fe080cb415ac8e63f0f5a2f466fc36ad87338e02759e00d1a2b"} Nov 03 20:41:29.074697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:29.074647 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" event={"ID":"90da652a-1f80-4e01-bf1d-7ef8a5f004ea","Type":"ContainerStarted","Data":"c86976ba8ee5fffa5767563c0e8ae3716f6a3836d4d6f0152e9e670012ba4cf2"} Nov 03 20:41:29.095982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:29.095923 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" podStartSLOduration=3.095907643 podStartE2EDuration="3.095907643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:29.094615306 +0000 UTC m=+2398.624961001" watchObservedRunningTime="2026-11-03 20:41:29.095907643 +0000 UTC m=+2398.626253315" Nov 03 20:41:29.184879 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:29.184850 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-5797404fedc5: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:29.184879 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:29.184876 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:29.185090 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:29.184939 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:31.184923447 +0000 UTC m=+2400.715269097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : secret "agc-cred-5797404fedc5" not found Nov 03 20:41:29.286626 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:29.286595 5815 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:29.286801 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:29.286679 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:31.286659069 +0000 UTC m=+2400.817004733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:30.026610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.026572 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr"] Nov 03 20:41:30.032670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.032644 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95772dc-4gsjr"] Nov 03 20:41:30.094606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.094573 5815 generic.go:356] "Generic (PLEG): container finished" podID="90da652a-1f80-4e01-bf1d-7ef8a5f004ea" containerID="107a14794a1b1fe080cb415ac8e63f0f5a2f466fc36ad87338e02759e00d1a2b" exitCode=0 Nov 03 20:41:30.094999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.094662 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" event={"ID":"90da652a-1f80-4e01-bf1d-7ef8a5f004ea","Type":"ContainerDied","Data":"107a14794a1b1fe080cb415ac8e63f0f5a2f466fc36ad87338e02759e00d1a2b"} Nov 03 20:41:30.563227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.563173 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" podUID="a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" containerName="payload-processing" containerID="cri-o://31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0" gracePeriod=25 Nov 03 20:41:30.592017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.591987 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd1f003-df61-4b5a-b5db-5dd8c493f860" path="/var/lib/kubelet/pods/bfd1f003-df61-4b5a-b5db-5dd8c493f860/volumes" Nov 03 20:41:30.881839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.881809 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:41:30.921800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.921767 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume\") pod \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " Nov 03 20:41:30.921972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.921837 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q\") pod \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " Nov 03 20:41:30.921972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.921919 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials\") pod \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " Nov 03 20:41:30.922106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.922052 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume\") pod \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\" (UID: \"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6\") " Nov 03 20:41:30.922215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.922187 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume" pod "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" (UID: "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:30.922425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.922399 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume" pod "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" (UID: "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:30.922835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.922815 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:30.922943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.922841 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:30.924295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.924270 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q" pod "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" (UID: "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6"). InnerVolumeSpecName "kube-api-access-jzq5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:30.924943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:30.924924 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials" pod "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" (UID: "a01591ce-c3f6-431a-b70f-bf5a0dcb40c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:31.025434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.025394 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzq5q\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-kube-api-access-jzq5q\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:31.025434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.025432 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:31.107295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.107200 5815 generic.go:356] "Generic (PLEG): container finished" podID="a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" containerID="31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0" exitCode=0 Nov 03 20:41:31.107295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.107255 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" Nov 03 20:41:31.107295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.107251 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" event={"ID":"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6","Type":"ContainerDied","Data":"31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0"} Nov 03 20:41:31.107960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.107409 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh" event={"ID":"a01591ce-c3f6-431a-b70f-bf5a0dcb40c6","Type":"ContainerDied","Data":"ac7102a14b965d3ec4c7a1d78d27093f8d12d11253b61e6d2051c1974eb2a06b"} Nov 03 20:41:31.107960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.107445 5815 scope.go:122] "RemoveContainer" containerID="31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0" Nov 03 20:41:31.123263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.123240 5815 scope.go:122] "RemoveContainer" containerID="31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0" Nov 03 20:41:31.123636 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:31.123608 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0\": container with ID starting with 31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0 not found: ID does not exist" containerID="31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0" Nov 03 20:41:31.123758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.123649 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0"} err="failed to get container status \"31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0\": rpc error: code = NotFound desc = could not find container \"31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0\": container with ID starting with 31a1377706c194090c7e0880b7bebdf747bebea4a0f389390db3466d198021a0 not found: ID does not exist" Nov 03 20:41:31.217639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.217599 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh"] Nov 03 20:41:31.227169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.227130 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-6454mscdh"] Nov 03 20:41:31.229045 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:31.228586 5815 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-5797404fedc5: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:31.229045 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:31.228610 5815 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8: secret "agc-cred-5797404fedc5" not found Nov 03 20:41:31.229045 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:31.228689 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:35.228668728 +0000 UTC m=+2404.759014398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : secret "agc-cred-5797404fedc5" not found Nov 03 20:41:31.331499 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:31.331471 5815 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:31.331706 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:31.331583 5815 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume podName:45364c34-1fee-4841-9d6f-15818e90d572 nodeName:}" failed. No retries permitted until 2026-11-03 20:41:35.331567437 +0000 UTC m=+2404.861913114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" (UID: "45364c34-1fee-4841-9d6f-15818e90d572") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found Nov 03 20:41:31.345215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.345191 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:31.378052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.377941 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:41:31.380569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.380548 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:41:31.432884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.432853 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca\") pod \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " Nov 03 20:41:31.433084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.432953 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt\" (UniqueName: \"kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt\") pod \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " Nov 03 20:41:31.433084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.432982 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca\") pod \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\" (UID: \"90da652a-1f80-4e01-bf1d-7ef8a5f004ea\") " Nov 03 20:41:31.434609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.434584 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca" pod "90da652a-1f80-4e01-bf1d-7ef8a5f004ea" (UID: "90da652a-1f80-4e01-bf1d-7ef8a5f004ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:31.435435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.435391 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:31.440174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.440147 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca" pod "90da652a-1f80-4e01-bf1d-7ef8a5f004ea" (UID: "90da652a-1f80-4e01-bf1d-7ef8a5f004ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:41:31.446953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.446865 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt" pod "90da652a-1f80-4e01-bf1d-7ef8a5f004ea" (UID: "90da652a-1f80-4e01-bf1d-7ef8a5f004ea"). InnerVolumeSpecName "kube-api-access-j6bvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:31.537128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.537093 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bvt\" (UniqueName: \"kubernetes.io/projected/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-kube-api-access-j6bvt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:31.537128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:31.537124 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90da652a-1f80-4e01-bf1d-7ef8a5f004ea-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:32.118319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:32.118288 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" Nov 03 20:41:32.118319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:32.118313 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg" event={"ID":"90da652a-1f80-4e01-bf1d-7ef8a5f004ea","Type":"ContainerDied","Data":"c86976ba8ee5fffa5767563c0e8ae3716f6a3836d4d6f0152e9e670012ba4cf2"} Nov 03 20:41:32.118794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:32.118362 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86976ba8ee5fffa5767563c0e8ae3716f6a3836d4d6f0152e9e670012ba4cf2" Nov 03 20:41:32.590862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:32.590827 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" path="/var/lib/kubelet/pods/a01591ce-c3f6-431a-b70f-bf5a0dcb40c6/volumes" Nov 03 20:41:33.014670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.014626 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" podUID="45364c34-1fee-4841-9d6f-15818e90d572" containerName="payload-processing" containerID="cri-o://6023ddf922b6193f189db5b6960d4a8dc64e2dcd5272c79c11b656a0c7fd7dc1" gracePeriod=25 Nov 03 20:41:33.133571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.133543 5815 generic.go:356] "Generic (PLEG): container finished" podID="45364c34-1fee-4841-9d6f-15818e90d572" containerID="6023ddf922b6193f189db5b6960d4a8dc64e2dcd5272c79c11b656a0c7fd7dc1" exitCode=0 Nov 03 20:41:33.133994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.133541 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" event={"ID":"45364c34-1fee-4841-9d6f-15818e90d572","Type":"ContainerDied","Data":"6023ddf922b6193f189db5b6960d4a8dc64e2dcd5272c79c11b656a0c7fd7dc1"} Nov 03 20:41:33.335245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.335215 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:33.387659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.387624 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6\") pod \"45364c34-1fee-4841-9d6f-15818e90d572\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " Nov 03 20:41:33.387873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.387705 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume\") pod \"45364c34-1fee-4841-9d6f-15818e90d572\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " Nov 03 20:41:33.387873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.387860 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials\") pod \"45364c34-1fee-4841-9d6f-15818e90d572\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " Nov 03 20:41:33.388144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.388124 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume\") pod \"45364c34-1fee-4841-9d6f-15818e90d572\" (UID: \"45364c34-1fee-4841-9d6f-15818e90d572\") " Nov 03 20:41:33.388241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.388134 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume" pod "45364c34-1fee-4841-9d6f-15818e90d572" (UID: "45364c34-1fee-4841-9d6f-15818e90d572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:33.388495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.388468 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume" pod "45364c34-1fee-4841-9d6f-15818e90d572" (UID: "45364c34-1fee-4841-9d6f-15818e90d572"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:33.388993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.388963 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:33.389095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.389000 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45364c34-1fee-4841-9d6f-15818e90d572-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:33.391271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.391243 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials" pod "45364c34-1fee-4841-9d6f-15818e90d572" (UID: "45364c34-1fee-4841-9d6f-15818e90d572"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:33.391405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.391290 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6" pod "45364c34-1fee-4841-9d6f-15818e90d572" (UID: "45364c34-1fee-4841-9d6f-15818e90d572"). InnerVolumeSpecName "kube-api-access-c5cz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:33.490905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.490872 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cz6\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-kube-api-access-c5cz6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:33.490905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:33.490901 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45364c34-1fee-4841-9d6f-15818e90d572-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:34.144848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:34.144811 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" event={"ID":"45364c34-1fee-4841-9d6f-15818e90d572","Type":"ContainerDied","Data":"6e90aa063563ee56f486a34b28ef3c094f9f06947affe34d4684939fb91d456e"} Nov 03 20:41:34.145307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:34.144878 5815 scope.go:122] "RemoveContainer" containerID="6023ddf922b6193f189db5b6960d4a8dc64e2dcd5272c79c11b656a0c7fd7dc1" Nov 03 20:41:34.145307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:34.144824 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8" Nov 03 20:41:34.175547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:34.175486 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8"] Nov 03 20:41:34.181291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:34.181266 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57d872zn8"] Nov 03 20:41:34.588875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:34.588839 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45364c34-1fee-4841-9d6f-15818e90d572" path="/var/lib/kubelet/pods/45364c34-1fee-4841-9d6f-15818e90d572/volumes" Nov 03 20:41:35.141031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:35.140996 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:41:35.142075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:35.142053 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:41:35.147236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:35.147213 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:41:35.148197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:35.148178 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:41:38.007534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:37.928157 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q"] Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.111923 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.111954 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.112035 5815 cpu_manager.go:414] "containerMap: removing container" podUID="90da652a-1f80-4e01-bf1d-7ef8a5f004ea" containerName="revoke-keys" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.112043 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90da652a-1f80-4e01-bf1d-7ef8a5f004ea" containerName="revoke-keys" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.112095 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7ba63e1d-67bc-40c3-ba3d-fad7fa505134" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.112104 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba63e1d-67bc-40c3-ba3d-fad7fa505134" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.112185 5815 cpu_manager.go:414] "containerMap: removing container" podUID="45364c34-1fee-4841-9d6f-15818e90d572" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.112193 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45364c34-1fee-4841-9d6f-15818e90d572" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.113368 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="90da652a-1f80-4e01-bf1d-7ef8a5f004ea" containerName="revoke-keys" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.113409 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba63e1d-67bc-40c3-ba3d-fad7fa505134" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.113434 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="45364c34-1fee-4841-9d6f-15818e90d572" containerName="payload-processing" Nov 03 20:41:38.120597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.113459 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01591ce-c3f6-431a-b70f-bf5a0dcb40c6" containerName="payload-processing" Nov 03 20:41:38.152290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.152259 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q"] Nov 03 20:41:38.152559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.152492 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.247601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.247562 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.247601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.247613 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.247826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.247733 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgh9\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.247826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.247765 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.349412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.349335 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.349412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.349373 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.349703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.349429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgh9\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.349703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.349468 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.350251 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.350223 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.350366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.350287 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.352115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.352094 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.362497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.362473 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgh9\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9\") pod \"payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:38.487757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:38.476810 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:39.043973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:39.038095 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q"] Nov 03 20:41:39.275706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:39.275642 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j"] Nov 03 20:41:39.286539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:39.286487 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" event={"ID":"384a2896-3da8-452c-ab1c-6135416e2093","Type":"ContainerStarted","Data":"b0d875d19ff784c455a7f1d396b047255420961c695bea77f7dfa276c511ae47"} Nov 03 20:41:39.319668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:39.319631 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q"] Nov 03 20:41:40.298838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:40.298801 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" event={"ID":"384a2896-3da8-452c-ab1c-6135416e2093","Type":"ContainerStarted","Data":"97e8049708ffe259da9d71ae44f8329e9551a7804c26a3139d0c4d0ad2cf1e8c"} Nov 03 20:41:40.299272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:40.298869 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:40.322752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:40.322705 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" podStartSLOduration=3.32269192 podStartE2EDuration="3.32269192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:40.31970552 +0000 UTC m=+2409.850051207" watchObservedRunningTime="2026-11-03 20:41:40.32269192 +0000 UTC m=+2409.853037617" Nov 03 20:41:41.031623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:41.031591 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279"] Nov 03 20:41:41.036732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:41.036704 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a217c30-xj279"] Nov 03 20:41:43.013765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:43.013723 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0425a08-2b24-443d-8d3f-a182414d94ed" path="/var/lib/kubelet/pods/b0425a08-2b24-443d-8d3f-a182414d94ed/volumes" Nov 03 20:41:44.084992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.084957 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2"] Nov 03 20:41:44.293377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.279153 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" podUID="3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" containerName="payload-processing" containerID="cri-o://7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340" gracePeriod=25 Nov 03 20:41:44.306539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.296195 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.325534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.310473 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.326249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.326221 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.326740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.326714 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bbv\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.327056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.327034 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.328030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.328008 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"routing-overlay\"" Nov 03 20:41:44.328242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.328226 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c7aecb\"" Nov 03 20:41:44.328431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.328417 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"kube-root-ca.crt\"" Nov 03 20:41:44.328655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.328641 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"openshift-service-ca.crt\"" Nov 03 20:41:44.350627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.328894 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"payload-processing-external-model-e2e--37db5ca3-dockercfg-d97nx\"" Nov 03 20:41:44.351540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.343226 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"agc-cred-65165f0977f9\"" Nov 03 20:41:44.477672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.430743 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.477672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.430800 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.477672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.430838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.477672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.431135 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bbv\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.477672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.432659 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.477672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.455571 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.490021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.489401 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:41:44.530830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.502049 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.530830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.519275 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2"] Nov 03 20:41:44.566294 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:44.566262 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:41:44.568315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.568282 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bbv\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.799788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.799761 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:44.842988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.842956 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume\") pod \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " Nov 03 20:41:44.843140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.842997 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials\") pod \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " Nov 03 20:41:44.843140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.843033 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume\") pod \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " Nov 03 20:41:44.843282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.843150 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn\") pod \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\" (UID: \"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5\") " Nov 03 20:41:44.843630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.843410 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume" pod "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" (UID: "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:44.843630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.843593 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume" pod "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" (UID: "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:44.844475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.844448 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:44.844754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.844479 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:44.845752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.845727 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials" pod "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" (UID: "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:44.845905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.845884 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn" pod "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" (UID: "3ca4552a-aa7f-48ae-a92e-e60ca97efcd5"). InnerVolumeSpecName "kube-api-access-822gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:44.865000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.864912 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:44.946181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.946102 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:44.946181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:44.946134 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822gn\" (UniqueName: \"kubernetes.io/projected/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5-kube-api-access-822gn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:45.157557 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:45.157528 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cc2d2e_8cbd_483d_94d6_bb9dcef234f0.slice/crio-3fbaf155f6dbc61141990ef2f9acb03fc18b90b6e37e6ff480485c363980619b WatchSource:0}: Error finding container 3fbaf155f6dbc61141990ef2f9acb03fc18b90b6e37e6ff480485c363980619b: Status 404 returned error can't find the container with id 3fbaf155f6dbc61141990ef2f9acb03fc18b90b6e37e6ff480485c363980619b Nov 03 20:41:45.158371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.158299 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2"] Nov 03 20:41:45.306532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.300623 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" podUID="384a2896-3da8-452c-ab1c-6135416e2093" containerName="payload-processing" containerID="cri-o://97e8049708ffe259da9d71ae44f8329e9551a7804c26a3139d0c4d0ad2cf1e8c" gracePeriod=25 Nov 03 20:41:45.581369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.581325 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" event={"ID":"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0","Type":"ContainerStarted","Data":"478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62"} Nov 03 20:41:45.581611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.581391 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:45.581611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.581412 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" event={"ID":"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0","Type":"ContainerStarted","Data":"3fbaf155f6dbc61141990ef2f9acb03fc18b90b6e37e6ff480485c363980619b"} Nov 03 20:41:45.582974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.582947 5815 generic.go:356] "Generic (PLEG): container finished" podID="3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" containerID="7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340" exitCode=0 Nov 03 20:41:45.583110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.583004 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" Nov 03 20:41:45.583110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.583024 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" event={"ID":"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5","Type":"ContainerDied","Data":"7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340"} Nov 03 20:41:45.583110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.583068 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j" event={"ID":"3ca4552a-aa7f-48ae-a92e-e60ca97efcd5","Type":"ContainerDied","Data":"3c09a05ca49735fa3c063ff5da472b21dd2781bbc2d4951dd5ed7eebe8b2d209"} Nov 03 20:41:45.583110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.583092 5815 scope.go:122] "RemoveContainer" containerID="7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340" Nov 03 20:41:45.585390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.585361 5815 generic.go:356] "Generic (PLEG): container finished" podID="384a2896-3da8-452c-ab1c-6135416e2093" containerID="97e8049708ffe259da9d71ae44f8329e9551a7804c26a3139d0c4d0ad2cf1e8c" exitCode=0 Nov 03 20:41:45.585656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.585631 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" event={"ID":"384a2896-3da8-452c-ab1c-6135416e2093","Type":"ContainerDied","Data":"97e8049708ffe259da9d71ae44f8329e9551a7804c26a3139d0c4d0ad2cf1e8c"} Nov 03 20:41:45.603718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.603689 5815 scope.go:122] "RemoveContainer" containerID="7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340" Nov 03 20:41:45.604150 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:41:45.604125 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340\": container with ID starting with 7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340 not found: ID does not exist" containerID="7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340" Nov 03 20:41:45.604322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.604294 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340"} err="failed to get container status \"7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340\": rpc error: code = NotFound desc = could not find container \"7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340\": container with ID starting with 7a4c8ab439c65947d59aefac2fd4e28abd5cbce0107ff354d94179770e4fa340 not found: ID does not exist" Nov 03 20:41:45.604783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.604733 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" podStartSLOduration=2.604715691 podStartE2EDuration="2.604715691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:45.60386431 +0000 UTC m=+2415.134209982" watchObservedRunningTime="2026-11-03 20:41:45.604715691 +0000 UTC m=+2415.135061367" Nov 03 20:41:45.635361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.626982 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j"] Nov 03 20:41:45.635361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.635266 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7f7fvt57j"] Nov 03 20:41:45.703719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.703656 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.786166 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume\") pod \"384a2896-3da8-452c-ab1c-6135416e2093\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.786269 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume\") pod \"384a2896-3da8-452c-ab1c-6135416e2093\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.786477 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9\") pod \"384a2896-3da8-452c-ab1c-6135416e2093\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.786629 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials\") pod \"384a2896-3da8-452c-ab1c-6135416e2093\" (UID: \"384a2896-3da8-452c-ab1c-6135416e2093\") " Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.787236 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume" pod "384a2896-3da8-452c-ab1c-6135416e2093" (UID: "384a2896-3da8-452c-ab1c-6135416e2093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.787529 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:45.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.788073 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume" pod "384a2896-3da8-452c-ab1c-6135416e2093" (UID: "384a2896-3da8-452c-ab1c-6135416e2093"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:41:45.812195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.810025 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials" pod "384a2896-3da8-452c-ab1c-6135416e2093" (UID: "384a2896-3da8-452c-ab1c-6135416e2093"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:45.819578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.816717 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9" pod "384a2896-3da8-452c-ab1c-6135416e2093" (UID: "384a2896-3da8-452c-ab1c-6135416e2093"). InnerVolumeSpecName "kube-api-access-6mgh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:41:45.889597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.889572 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mgh9\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-kube-api-access-6mgh9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:45.889597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.889595 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/384a2896-3da8-452c-ab1c-6135416e2093-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:45.889790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:45.889605 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/384a2896-3da8-452c-ab1c-6135416e2093-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:41:46.590792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:46.590756 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" path="/var/lib/kubelet/pods/3ca4552a-aa7f-48ae-a92e-e60ca97efcd5/volumes" Nov 03 20:41:46.597207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:46.597176 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" Nov 03 20:41:46.597422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:46.597204 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q" event={"ID":"384a2896-3da8-452c-ab1c-6135416e2093","Type":"ContainerDied","Data":"b0d875d19ff784c455a7f1d396b047255420961c695bea77f7dfa276c511ae47"} Nov 03 20:41:46.597851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:46.597822 5815 scope.go:122] "RemoveContainer" containerID="97e8049708ffe259da9d71ae44f8329e9551a7804c26a3139d0c4d0ad2cf1e8c" Nov 03 20:41:46.625591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:46.625561 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q"] Nov 03 20:41:46.631218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:46.631197 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14ebbb/payload-processing-external-model-e2e-extmod-a-14ebbb-7bf84tm4q"] Nov 03 20:41:48.587644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:48.587613 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384a2896-3da8-452c-ab1c-6135416e2093" path="/var/lib/kubelet/pods/384a2896-3da8-452c-ab1c-6135416e2093/volumes" Nov 03 20:41:49.513976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.513944 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7"] Nov 03 20:41:49.514725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.514710 5815 cpu_manager.go:414] "containerMap: removing container" podUID="384a2896-3da8-452c-ab1c-6135416e2093" containerName="payload-processing" Nov 03 20:41:49.514725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.514725 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384a2896-3da8-452c-ab1c-6135416e2093" containerName="payload-processing" Nov 03 20:41:49.514824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.514764 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" containerName="payload-processing" Nov 03 20:41:49.514824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.514770 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" containerName="payload-processing" Nov 03 20:41:49.514895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.514890 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca4552a-aa7f-48ae-a92e-e60ca97efcd5" containerName="payload-processing" Nov 03 20:41:49.514933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.514906 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="384a2896-3da8-452c-ab1c-6135416e2093" containerName="payload-processing" Nov 03 20:41:49.520204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.520185 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.528341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.528309 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-98870d\"/\"e2e-usage-sim-b9c8f1-cert\"" Nov 03 20:41:49.528485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.528427 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-98870d\"/\"default-dockercfg-vv6js\"" Nov 03 20:41:49.528583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.528567 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-98870d\"/\"kube-root-ca.crt\"" Nov 03 20:41:49.528653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.528625 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-98870d\"/\"openshift-service-ca.crt\"" Nov 03 20:41:49.531352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.531325 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7"] Nov 03 20:41:49.593361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.593330 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls\") pod \"e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.593756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.593378 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjrn6\" (UniqueName: \"kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6\") pod \"e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.698955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.698911 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls\") pod \"e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.699799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.699770 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjrn6\" (UniqueName: \"kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6\") pod \"e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.736775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.736699 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjrn6\" (UniqueName: \"kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6\") pod \"e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.736968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.736942 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls\") pod \"e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:49.853085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:49.853008 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:50.145971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:50.145942 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7"] Nov 03 20:41:50.146224 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:50.146200 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ad21e0b_47dd_47c8_8e58_623a40229bb4.slice/crio-674e17884f3637e44221795349f16ded6ae61129fe957bcfc49d7842b698a05e WatchSource:0}: Error finding container 674e17884f3637e44221795349f16ded6ae61129fe957bcfc49d7842b698a05e: Status 404 returned error can't find the container with id 674e17884f3637e44221795349f16ded6ae61129fe957bcfc49d7842b698a05e Nov 03 20:41:50.662616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:50.662577 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" event={"ID":"1ad21e0b-47dd-47c8-8e58-623a40229bb4","Type":"ContainerStarted","Data":"72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30"} Nov 03 20:41:50.663136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:50.662626 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" event={"ID":"1ad21e0b-47dd-47c8-8e58-623a40229bb4","Type":"ContainerStarted","Data":"674e17884f3637e44221795349f16ded6ae61129fe957bcfc49d7842b698a05e"} Nov 03 20:41:50.663136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:50.662648 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:50.668124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:50.668098 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:41:50.689360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:50.689307 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" podStartSLOduration=1.68929551 podStartE2EDuration="1.68929551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:50.682254425 +0000 UTC m=+2420.212600099" watchObservedRunningTime="2026-11-03 20:41:50.68929551 +0000 UTC m=+2420.219641201" Nov 03 20:41:54.387750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.387716 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-598bd85469-hbkd8"] Nov 03 20:41:54.397009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.396981 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.402928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.402446 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:41:54.402928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.402722 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:41:54.402928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.402764 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:41:54.402928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.402819 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1ffbdf40cba2\"" Nov 03 20:41:54.402928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.402879 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:41:54.404624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.404587 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-t2q4s\"" Nov 03 20:41:54.408297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.408258 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-598bd85469-hbkd8"] Nov 03 20:41:54.496599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.496561 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.496967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.496941 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.497116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.497101 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bcm\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.497345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.497326 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.607908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.607859 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.608312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.608267 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.608643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.608622 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.608880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.608861 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bcm\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.635536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.610428 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.637716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.636316 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.649575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.644460 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.655934 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.655849 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bcm\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm\") pod \"payload-processing-external-model-598bd85469-hbkd8\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:54.736034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:54.735130 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:55.252592 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:55.252560 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b69c13d_cea1_4e05_a9fa_c22dcc07f852.slice/crio-1fb14d1c4bf3cc859250f3b054520cc31fa14470e637c8154de4f875bbb32898 WatchSource:0}: Error finding container 1fb14d1c4bf3cc859250f3b054520cc31fa14470e637c8154de4f875bbb32898: Status 404 returned error can't find the container with id 1fb14d1c4bf3cc859250f3b054520cc31fa14470e637c8154de4f875bbb32898 Nov 03 20:41:55.252824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:55.252804 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-598bd85469-hbkd8"] Nov 03 20:41:56.073474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.073416 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" event={"ID":"6b69c13d-cea1-4e05-a9fa-c22dcc07f852","Type":"ContainerStarted","Data":"67e5293f62df685277d729b4c54a476c4e8c267be1cce953f8aaf93a2016ad92"} Nov 03 20:41:56.074121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.074093 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" event={"ID":"6b69c13d-cea1-4e05-a9fa-c22dcc07f852","Type":"ContainerStarted","Data":"1fb14d1c4bf3cc859250f3b054520cc31fa14470e637c8154de4f875bbb32898"} Nov 03 20:41:56.079546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.075589 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:41:56.113765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.113704 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" podStartSLOduration=2.113683367 podStartE2EDuration="2.113683367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:56.113147286 +0000 UTC m=+2425.643492960" watchObservedRunningTime="2026-11-03 20:41:56.113683367 +0000 UTC m=+2425.644029042" Nov 03 20:41:56.527632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.527586 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5"] Nov 03 20:41:56.563672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.558561 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.570087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.570060 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-8c680bc0dfe3\"" Nov 03 20:41:56.571026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.571000 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-e2e--0dce6837-dockercfg-8c4ts\"" Nov 03 20:41:56.571232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.571216 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-233866678b02\"" Nov 03 20:41:56.573236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.573216 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"routing-overlay\"" Nov 03 20:41:56.573468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.573442 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"openshift-service-ca.crt\"" Nov 03 20:41:56.573715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.573700 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"kube-root-ca.crt\"" Nov 03 20:41:56.573922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.573906 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-plugins-e2e-extmod-a-37aa95\"" Nov 03 20:41:56.669426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.668132 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czvb5\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.669426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.668337 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.669426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.668453 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.669426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.668491 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.742177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.740957 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5"] Nov 03 20:41:56.742177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.741139 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:41:56.764057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.764031 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w"] Nov 03 20:41:56.774924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.774605 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czvb5\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.775449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.775426 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.775737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.775717 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.775901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.775884 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.783021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.782960 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.805369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.805315 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.816297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.816259 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czvb5\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.819725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.819699 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.821002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.820980 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:56.824887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.824867 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w"] Nov 03 20:41:56.825246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.825230 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"agc-cred-75124d678115\"" Nov 03 20:41:56.878107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.877989 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.878267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.878226 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.878347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.878295 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9jn\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.878566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.878547 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.984809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.980955 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.984809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.981025 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9jn\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.984809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.981183 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.984809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.981470 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:56.986771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:56.986741 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:57.008367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.008145 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:57.009031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.009003 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:57.012395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.012368 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:57.024292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.024262 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9jn\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:57.161197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.161168 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:57.323029 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:57.323001 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fcacca_d0b1_4d6e_9729_0a92786838c6.slice/crio-30f5142d0f02411edd11dc2d0715c126bed4bba276841f921b906527027643fe WatchSource:0}: Error finding container 30f5142d0f02411edd11dc2d0715c126bed4bba276841f921b906527027643fe: Status 404 returned error can't find the container with id 30f5142d0f02411edd11dc2d0715c126bed4bba276841f921b906527027643fe Nov 03 20:41:57.326598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.326576 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5"] Nov 03 20:41:57.473165 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:41:57.473135 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a4d719_b8cc_4080_910a_ede38831cbfe.slice/crio-b33252742dc378e66c0d22cca54ec34db5a6f4725c3edf34f865980a1205c008 WatchSource:0}: Error finding container b33252742dc378e66c0d22cca54ec34db5a6f4725c3edf34f865980a1205c008: Status 404 returned error can't find the container with id b33252742dc378e66c0d22cca54ec34db5a6f4725c3edf34f865980a1205c008 Nov 03 20:41:57.474443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:57.474339 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w"] Nov 03 20:41:58.097918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.097881 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" event={"ID":"74a4d719-b8cc-4080-910a-ede38831cbfe","Type":"ContainerStarted","Data":"acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4"} Nov 03 20:41:58.098097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.097945 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" event={"ID":"74a4d719-b8cc-4080-910a-ede38831cbfe","Type":"ContainerStarted","Data":"b33252742dc378e66c0d22cca54ec34db5a6f4725c3edf34f865980a1205c008"} Nov 03 20:41:58.098097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.097973 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:41:58.100247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.100219 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" event={"ID":"33fcacca-d0b1-4d6e-9729-0a92786838c6","Type":"ContainerStarted","Data":"0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5"} Nov 03 20:41:58.100389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.100265 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" event={"ID":"33fcacca-d0b1-4d6e-9729-0a92786838c6","Type":"ContainerStarted","Data":"30f5142d0f02411edd11dc2d0715c126bed4bba276841f921b906527027643fe"} Nov 03 20:41:58.100389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.100294 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:41:58.119818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.119774 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" podStartSLOduration=2.119761766 podStartE2EDuration="2.119761766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:58.118314417 +0000 UTC m=+2427.648660090" watchObservedRunningTime="2026-11-03 20:41:58.119761766 +0000 UTC m=+2427.650107437" Nov 03 20:41:58.144050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:41:58.144006 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" podStartSLOduration=2.1439931469999998 podStartE2EDuration="2.143993147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:58.140495348 +0000 UTC m=+2427.670841021" watchObservedRunningTime="2026-11-03 20:41:58.143993147 +0000 UTC m=+2427.674338820" Nov 03 20:42:00.987528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:00.961851 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:42:01.236530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:01.226080 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:01.426715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:01.426607 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:01.459639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:01.459592 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:01.747571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:01.738604 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:01.822049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:01.822008 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:07.086998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:07.086968 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:42:09.114193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:09.114116 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:42:09.114193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:09.114181 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:42:09.185408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:09.185371 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2"] Nov 03 20:42:13.854049 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:42:13.854014 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:42:14.186560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.186437 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" podUID="f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" containerName="payload-processing" containerID="cri-o://478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62" gracePeriod=25 Nov 03 20:42:14.539191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.539166 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:42:14.631669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.631640 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume\") pod \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " Nov 03 20:42:14.631848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.631676 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv\") pod \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " Nov 03 20:42:14.631848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.631812 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials\") pod \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " Nov 03 20:42:14.632108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.632080 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume" pod "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" (UID: "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:14.632187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.632135 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume\") pod \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\" (UID: \"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0\") " Nov 03 20:42:14.632501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.632470 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume" pod "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" (UID: "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:42:14.632816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.632795 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:14.632918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.632821 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:14.634299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.634272 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv" pod "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" (UID: "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0"). InnerVolumeSpecName "kube-api-access-58bbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:14.634647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.634628 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials" pod "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" (UID: "f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:42:14.658835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.658807 5815 generic.go:356] "Generic (PLEG): container finished" podID="f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" containerID="478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62" exitCode=0 Nov 03 20:42:14.658935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.658857 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" Nov 03 20:42:14.658935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.658883 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" event={"ID":"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0","Type":"ContainerDied","Data":"478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62"} Nov 03 20:42:14.659018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.658936 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2" event={"ID":"f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0","Type":"ContainerDied","Data":"3fbaf155f6dbc61141990ef2f9acb03fc18b90b6e37e6ff480485c363980619b"} Nov 03 20:42:14.659018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.658958 5815 scope.go:122] "RemoveContainer" containerID="478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62" Nov 03 20:42:14.691804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.691781 5815 scope.go:122] "RemoveContainer" containerID="478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62" Nov 03 20:42:14.692175 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:42:14.692153 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62\": container with ID starting with 478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62 not found: ID does not exist" containerID="478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62" Nov 03 20:42:14.692231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.692188 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62"} err="failed to get container status \"478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62\": rpc error: code = NotFound desc = could not find container \"478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62\": container with ID starting with 478e3047e7bee56fa81f1111245b432fb4f731346478692626685b8574d85e62 not found: ID does not exist" Nov 03 20:42:14.708602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.708576 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2"] Nov 03 20:42:14.717427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.717400 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96z2wp2"] Nov 03 20:42:14.734481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.734440 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58bbv\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-kube-api-access-58bbv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:14.734481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:14.734478 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:42:16.588131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:16.588093 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" path="/var/lib/kubelet/pods/f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0/volumes" Nov 03 20:42:22.028937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:22.028902 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2"] Nov 03 20:42:22.034106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:22.034080 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0790b2f-g9cd2"] Nov 03 20:42:22.592276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:22.592242 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa" path="/var/lib/kubelet/pods/b6ef4b7e-a9d2-448a-9617-446d9fd1bbfa/volumes" Nov 03 20:42:35.360214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:35.360181 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:42:38.041127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:38.041060 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq"] Nov 03 20:42:38.052556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:38.052526 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f1b668-ql8sq"] Nov 03 20:42:38.651218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:38.651181 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da52e4d9-934b-4775-992c-18c2db92df7a" path="/var/lib/kubelet/pods/da52e4d9-934b-4775-992c-18c2db92df7a/volumes" Nov 03 20:42:40.434626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:40.434584 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:42:43.852983 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:42:43.852941 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:42:44.623946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:44.623907 5815 scope.go:122] "RemoveContainer" containerID="3de1c338cf428979b4e0c2077b422069ae5345b9bc9805248c240ff1d5400d95" Nov 03 20:42:44.635574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:44.635552 5815 scope.go:122] "RemoveContainer" containerID="71d2ba0face54bcda5f5eb57f0a4c805987f2f128eb1df4b2b761d7368b226bb" Nov 03 20:42:51.033103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:51.033062 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6"] Nov 03 20:42:51.038105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:51.038078 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3200e506-kv8f6"] Nov 03 20:42:52.590809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:52.590775 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3aef707-29ed-4249-9d05-11c637aefdfb" path="/var/lib/kubelet/pods/e3aef707-29ed-4249-9d05-11c637aefdfb/volumes" Nov 03 20:42:55.025338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:55.025302 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk"] Nov 03 20:42:55.030474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:55.030448 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d617d-kwwrk"] Nov 03 20:42:56.592748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:42:56.592715 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caaec364-a55c-4ca2-840c-05aa6f2a5a33" path="/var/lib/kubelet/pods/caaec364-a55c-4ca2-840c-05aa6f2a5a33/volumes" Nov 03 20:43:02.031894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:02.031856 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw"] Nov 03 20:43:02.036657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:02.036634 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3704215d-pzclw"] Nov 03 20:43:02.587870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:02.587840 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39e5864-04b8-4ee6-ba19-f933c274b34e" path="/var/lib/kubelet/pods/a39e5864-04b8-4ee6-ba19-f933c274b34e/volumes" Nov 03 20:43:12.286285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:12.286240 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:43:14.353531 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:43:14.349913 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:43:14.417597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:14.415075 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:14.498629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:14.498589 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:16.733533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:16.732596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:17.004592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:17.003636 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:17.214534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:17.213586 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:17.294602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:17.292138 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:17.457535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:17.454609 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:18.216534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:18.215567 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:18.216534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:18.216119 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:43:18.260161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:18.260124 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:43:20.464775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.464737 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w"] Nov 03 20:43:20.466646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.466614 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" containerName="payload-processing" Nov 03 20:43:20.466646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.466643 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" containerName="payload-processing" Nov 03 20:43:20.466866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.466852 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6cc2d2e-8cbd-483d-94d6-bb9dcef234f0" containerName="payload-processing" Nov 03 20:43:20.472979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.472956 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.483392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.483363 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:43:20.483570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.483432 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:43:20.487046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.487022 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w"] Nov 03 20:43:20.670534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.666192 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwd5\" (UniqueName: \"kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.670534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.666644 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.670534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.666856 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.777606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.772618 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.777606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.772773 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.777606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.772866 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwd5\" (UniqueName: \"kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.796527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.794012 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.934851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.934281 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:20.979531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:20.955603 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwd5\" (UniqueName: \"kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5\") pod \"maas-api-revoke-sub-3565bf14-kqf2w\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:21.107569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:21.107458 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:21.401988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:21.401954 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w"] Nov 03 20:43:21.402144 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:43:21.402046 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73eb9e7c_2ed0_4b75_b99b_b6630cd05925.slice/crio-4baab384846d208ed688464ab35794aac8566c8f738a2f72267aabd99ac4cf8c WatchSource:0}: Error finding container 4baab384846d208ed688464ab35794aac8566c8f738a2f72267aabd99ac4cf8c: Status 404 returned error can't find the container with id 4baab384846d208ed688464ab35794aac8566c8f738a2f72267aabd99ac4cf8c Nov 03 20:43:22.387829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:22.387793 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" event={"ID":"73eb9e7c-2ed0-4b75-b99b-b6630cd05925","Type":"ContainerStarted","Data":"e3a317220a3a7ab7c4011c1c752ceba7e275e695755835cbd7db4730e736d533"} Nov 03 20:43:22.388249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:22.387860 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" event={"ID":"73eb9e7c-2ed0-4b75-b99b-b6630cd05925","Type":"ContainerStarted","Data":"4baab384846d208ed688464ab35794aac8566c8f738a2f72267aabd99ac4cf8c"} Nov 03 20:43:22.407444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:22.407384 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" podStartSLOduration=2.407366507 podStartE2EDuration="2.407366507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:22.405476039 +0000 UTC m=+2511.935821712" watchObservedRunningTime="2026-11-03 20:43:22.407366507 +0000 UTC m=+2511.937712181" Nov 03 20:43:23.400654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.400616 5815 generic.go:356] "Generic (PLEG): container finished" podID="73eb9e7c-2ed0-4b75-b99b-b6630cd05925" containerID="e3a317220a3a7ab7c4011c1c752ceba7e275e695755835cbd7db4730e736d533" exitCode=0 Nov 03 20:43:23.401181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.400753 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" event={"ID":"73eb9e7c-2ed0-4b75-b99b-b6630cd05925","Type":"ContainerDied","Data":"e3a317220a3a7ab7c4011c1c752ceba7e275e695755835cbd7db4730e736d533"} Nov 03 20:43:23.762794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.762765 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj"] Nov 03 20:43:23.777773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.768355 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.777948 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.777813 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj"] Nov 03 20:43:23.892639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.892607 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznwc\" (UniqueName: \"kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.892829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.892687 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.892914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.892844 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.994943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.994909 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznwc\" (UniqueName: \"kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.995146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.994972 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.995146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.995069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.995720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.995694 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:23.997846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:23.997823 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:24.007338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.007310 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznwc\" (UniqueName: \"kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc\") pod \"maas-api-revoke-sub-0e136d4b-9xfhj\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:24.195664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.195580 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:24.488584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.488554 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj"] Nov 03 20:43:24.514610 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:43:24.514576 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ffdcfe_a821_4f1c_ab66_f35dddc89e28.slice/crio-8451afd7c5d4552be96d1e9a40ca06bb85fdd2a427bedc12c7206bcbc8ea0c49 WatchSource:0}: Error finding container 8451afd7c5d4552be96d1e9a40ca06bb85fdd2a427bedc12c7206bcbc8ea0c49: Status 404 returned error can't find the container with id 8451afd7c5d4552be96d1e9a40ca06bb85fdd2a427bedc12c7206bcbc8ea0c49 Nov 03 20:43:24.700701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.700674 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:24.828641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.828545 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca\") pod \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " Nov 03 20:43:24.828903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.828751 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca\") pod \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " Nov 03 20:43:24.828903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.828860 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5\" (UniqueName: \"kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5\") pod \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\" (UID: \"73eb9e7c-2ed0-4b75-b99b-b6630cd05925\") " Nov 03 20:43:24.829146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.829124 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca" pod "73eb9e7c-2ed0-4b75-b99b-b6630cd05925" (UID: "73eb9e7c-2ed0-4b75-b99b-b6630cd05925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:24.829631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.829608 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:24.865176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.865149 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca" pod "73eb9e7c-2ed0-4b75-b99b-b6630cd05925" (UID: "73eb9e7c-2ed0-4b75-b99b-b6630cd05925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:24.865263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.865190 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5" pod "73eb9e7c-2ed0-4b75-b99b-b6630cd05925" (UID: "73eb9e7c-2ed0-4b75-b99b-b6630cd05925"). InnerVolumeSpecName "kube-api-access-7kwd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:24.930871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.930848 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwd5\" (UniqueName: \"kubernetes.io/projected/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-kube-api-access-7kwd5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:24.930871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:24.930872 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73eb9e7c-2ed0-4b75-b99b-b6630cd05925-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:25.434637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:25.434612 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" Nov 03 20:43:25.434845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:25.434658 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w" event={"ID":"73eb9e7c-2ed0-4b75-b99b-b6630cd05925","Type":"ContainerDied","Data":"4baab384846d208ed688464ab35794aac8566c8f738a2f72267aabd99ac4cf8c"} Nov 03 20:43:25.434845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:25.434739 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4baab384846d208ed688464ab35794aac8566c8f738a2f72267aabd99ac4cf8c" Nov 03 20:43:25.436718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:25.436689 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" event={"ID":"86ffdcfe-a821-4f1c-ab66-f35dddc89e28","Type":"ContainerStarted","Data":"d41a30bfb4a273d5bde0800200fc647179a1d329b34dcb0aaa28d0c14b9c23c4"} Nov 03 20:43:25.436854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:25.436741 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" event={"ID":"86ffdcfe-a821-4f1c-ab66-f35dddc89e28","Type":"ContainerStarted","Data":"8451afd7c5d4552be96d1e9a40ca06bb85fdd2a427bedc12c7206bcbc8ea0c49"} Nov 03 20:43:25.460996 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:25.460943 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" podStartSLOduration=2.4609274230000002 podStartE2EDuration="2.460927423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:25.454420872 +0000 UTC m=+2514.984766545" watchObservedRunningTime="2026-11-03 20:43:25.460927423 +0000 UTC m=+2514.991273100" Nov 03 20:43:26.448024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:26.447987 5815 generic.go:356] "Generic (PLEG): container finished" podID="86ffdcfe-a821-4f1c-ab66-f35dddc89e28" containerID="d41a30bfb4a273d5bde0800200fc647179a1d329b34dcb0aaa28d0c14b9c23c4" exitCode=0 Nov 03 20:43:26.448420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:26.448073 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" event={"ID":"86ffdcfe-a821-4f1c-ab66-f35dddc89e28","Type":"ContainerDied","Data":"d41a30bfb4a273d5bde0800200fc647179a1d329b34dcb0aaa28d0c14b9c23c4"} Nov 03 20:43:27.740921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.740896 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:27.900743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.900655 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca\") pod \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " Nov 03 20:43:27.900743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.900717 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca\") pod \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " Nov 03 20:43:27.900995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.900948 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc\" (UniqueName: \"kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc\") pod \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\" (UID: \"86ffdcfe-a821-4f1c-ab66-f35dddc89e28\") " Nov 03 20:43:27.901058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.900979 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca" pod "86ffdcfe-a821-4f1c-ab66-f35dddc89e28" (UID: "86ffdcfe-a821-4f1c-ab66-f35dddc89e28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:27.901874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.901845 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:27.903209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.903183 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca" pod "86ffdcfe-a821-4f1c-ab66-f35dddc89e28" (UID: "86ffdcfe-a821-4f1c-ab66-f35dddc89e28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:43:27.903364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:27.903346 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc" pod "86ffdcfe-a821-4f1c-ab66-f35dddc89e28" (UID: "86ffdcfe-a821-4f1c-ab66-f35dddc89e28"). InnerVolumeSpecName "kube-api-access-zznwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:28.002895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:28.002862 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zznwc\" (UniqueName: \"kubernetes.io/projected/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-kube-api-access-zznwc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:28.002895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:28.002889 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86ffdcfe-a821-4f1c-ab66-f35dddc89e28-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:28.526667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:28.526635 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" Nov 03 20:43:28.526837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:28.526668 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj" event={"ID":"86ffdcfe-a821-4f1c-ab66-f35dddc89e28","Type":"ContainerDied","Data":"8451afd7c5d4552be96d1e9a40ca06bb85fdd2a427bedc12c7206bcbc8ea0c49"} Nov 03 20:43:28.526837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:28.526715 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8451afd7c5d4552be96d1e9a40ca06bb85fdd2a427bedc12c7206bcbc8ea0c49" Nov 03 20:43:31.336632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:31.336591 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v"] Nov 03 20:43:34.639946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:34.635500 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-598bd85469-hbkd8"] Nov 03 20:43:36.337671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.337628 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" podUID="b0989e9a-50f9-4762-8f6f-8119decc007d" containerName="payload-processing" containerID="cri-o://e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5" gracePeriod=25 Nov 03 20:43:36.649416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.649382 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:43:36.751318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.751257 5815 generic.go:356] "Generic (PLEG): container finished" podID="b0989e9a-50f9-4762-8f6f-8119decc007d" containerID="e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5" exitCode=0 Nov 03 20:43:36.751543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.751299 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" event={"ID":"b0989e9a-50f9-4762-8f6f-8119decc007d","Type":"ContainerDied","Data":"e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5"} Nov 03 20:43:36.751683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.751590 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" event={"ID":"b0989e9a-50f9-4762-8f6f-8119decc007d","Type":"ContainerDied","Data":"d54d4e442b53102052213105fc426fb5f075ecd19ff767031f9fcc9254c86bc6"} Nov 03 20:43:36.751683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.751322 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v" Nov 03 20:43:36.751683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.751626 5815 scope.go:122] "RemoveContainer" containerID="e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5" Nov 03 20:43:36.764235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.764215 5815 scope.go:122] "RemoveContainer" containerID="e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5" Nov 03 20:43:36.764571 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:43:36.764541 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5\": container with ID starting with e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5 not found: ID does not exist" containerID="e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5" Nov 03 20:43:36.764683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.764568 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5"} err="failed to get container status \"e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5\": rpc error: code = NotFound desc = could not find container \"e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5\": container with ID starting with e883aa91095c7d4b5977b6948b517703eea23c69e34621ea5b321adfb1e518a5 not found: ID does not exist" Nov 03 20:43:36.767296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.767263 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials\") pod \"b0989e9a-50f9-4762-8f6f-8119decc007d\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " Nov 03 20:43:36.767584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.767563 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume\") pod \"b0989e9a-50f9-4762-8f6f-8119decc007d\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " Nov 03 20:43:36.767681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.767605 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume\") pod \"b0989e9a-50f9-4762-8f6f-8119decc007d\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " Nov 03 20:43:36.767681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.767647 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq\") pod \"b0989e9a-50f9-4762-8f6f-8119decc007d\" (UID: \"b0989e9a-50f9-4762-8f6f-8119decc007d\") " Nov 03 20:43:36.769961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.768069 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume" pod "b0989e9a-50f9-4762-8f6f-8119decc007d" (UID: "b0989e9a-50f9-4762-8f6f-8119decc007d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:36.769961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.768185 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume" pod "b0989e9a-50f9-4762-8f6f-8119decc007d" (UID: "b0989e9a-50f9-4762-8f6f-8119decc007d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:36.769961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.768429 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:36.769961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.768450 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0989e9a-50f9-4762-8f6f-8119decc007d-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:36.770268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.770150 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials" pod "b0989e9a-50f9-4762-8f6f-8119decc007d" (UID: "b0989e9a-50f9-4762-8f6f-8119decc007d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:36.770395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.770375 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq" pod "b0989e9a-50f9-4762-8f6f-8119decc007d" (UID: "b0989e9a-50f9-4762-8f6f-8119decc007d"). InnerVolumeSpecName "kube-api-access-bn4pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:36.877212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.877045 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:36.880386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.879443 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn4pq\" (UniqueName: \"kubernetes.io/projected/b0989e9a-50f9-4762-8f6f-8119decc007d-kube-api-access-bn4pq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:36.998793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:36.998738 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m"] Nov 03 20:43:37.000399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000365 5815 cpu_manager.go:414] "containerMap: removing container" podUID="73eb9e7c-2ed0-4b75-b99b-b6630cd05925" containerName="revoke-keys" Nov 03 20:43:37.000399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000398 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73eb9e7c-2ed0-4b75-b99b-b6630cd05925" containerName="revoke-keys" Nov 03 20:43:37.000695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000460 5815 cpu_manager.go:414] "containerMap: removing container" podUID="86ffdcfe-a821-4f1c-ab66-f35dddc89e28" containerName="revoke-keys" Nov 03 20:43:37.000695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000469 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ffdcfe-a821-4f1c-ab66-f35dddc89e28" containerName="revoke-keys" Nov 03 20:43:37.000695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000544 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b0989e9a-50f9-4762-8f6f-8119decc007d" containerName="payload-processing" Nov 03 20:43:37.000695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000552 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0989e9a-50f9-4762-8f6f-8119decc007d" containerName="payload-processing" Nov 03 20:43:37.000950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000758 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0989e9a-50f9-4762-8f6f-8119decc007d" containerName="payload-processing" Nov 03 20:43:37.000950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000782 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="73eb9e7c-2ed0-4b75-b99b-b6630cd05925" containerName="revoke-keys" Nov 03 20:43:37.000950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.000801 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ffdcfe-a821-4f1c-ab66-f35dddc89e28" containerName="revoke-keys" Nov 03 20:43:37.023759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.023722 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m"] Nov 03 20:43:37.023928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.023845 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.031434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.031385 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3d4fe2-openshift-default-dockercfg-rxt6m\"" Nov 03 20:43:37.083315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083280 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nclqw\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.083494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083344 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.083494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083404 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.083669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083501 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.083669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083578 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.083669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083636 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.083953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083922 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.084051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.083996 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.084051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.084033 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.094067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.094043 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v"] Nov 03 20:43:37.102010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.101989 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d8d0fa/payload-processing-external-model-e2e-extmod-a-d8d0fa-8d97qbg8v"] Nov 03 20:43:37.226590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.221185 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.253748 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.253943 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nclqw\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.253993 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.254071 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.254206 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.254270 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.254344 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.254524 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.254571 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.255376 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.255972 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.259532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.256436 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.261417 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.261390 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.301530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.300628 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.309956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.302119 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.331779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.331749 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.342282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.342222 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclqw\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.366553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.366293 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:37.972635 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:43:37.972598 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a28dc7_b27e_4c27_97ee_38996ee945b5.slice/crio-2260d63b661491beedb7401b6b0c2db3d71821e44a2223d7692b4823d3d6c3df WatchSource:0}: Error finding container 2260d63b661491beedb7401b6b0c2db3d71821e44a2223d7692b4823d3d6c3df: Status 404 returned error can't find the container with id 2260d63b661491beedb7401b6b0c2db3d71821e44a2223d7692b4823d3d6c3df Nov 03 20:43:37.974224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.974198 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m"] Nov 03 20:43:37.976330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.976298 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:43:37.976425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.976363 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:43:37.976425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:37.976392 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:43:38.639443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:38.639356 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0989e9a-50f9-4762-8f6f-8119decc007d" path="/var/lib/kubelet/pods/b0989e9a-50f9-4762-8f6f-8119decc007d/volumes" Nov 03 20:43:38.778579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:38.778541 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" event={"ID":"b6a28dc7-b27e-4c27-97ee-38996ee945b5","Type":"ContainerStarted","Data":"02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1"} Nov 03 20:43:38.778728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:38.778606 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" event={"ID":"b6a28dc7-b27e-4c27-97ee-38996ee945b5","Type":"ContainerStarted","Data":"2260d63b661491beedb7401b6b0c2db3d71821e44a2223d7692b4823d3d6c3df"} Nov 03 20:43:38.803230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:38.803177 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podStartSLOduration=2.803161085 podStartE2EDuration="2.803161085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:38.802067668 +0000 UTC m=+2528.332413340" watchObservedRunningTime="2026-11-03 20:43:38.803161085 +0000 UTC m=+2528.333506758" Nov 03 20:43:39.367671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.367634 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:39.374069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.374042 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:39.640701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.640610 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" podUID="6b69c13d-cea1-4e05-a9fa-c22dcc07f852" containerName="payload-processing" containerID="cri-o://67e5293f62df685277d729b4c54a476c4e8c267be1cce953f8aaf93a2016ad92" gracePeriod=25 Nov 03 20:43:39.790990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.790959 5815 generic.go:356] "Generic (PLEG): container finished" podID="6b69c13d-cea1-4e05-a9fa-c22dcc07f852" containerID="67e5293f62df685277d729b4c54a476c4e8c267be1cce953f8aaf93a2016ad92" exitCode=0 Nov 03 20:43:39.791288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.791187 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" event={"ID":"6b69c13d-cea1-4e05-a9fa-c22dcc07f852","Type":"ContainerDied","Data":"67e5293f62df685277d729b4c54a476c4e8c267be1cce953f8aaf93a2016ad92"} Nov 03 20:43:39.791288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.791258 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:39.792489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.792470 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:43:39.976563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.976541 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:43:39.982005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.981983 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume\") pod \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " Nov 03 20:43:39.982122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982067 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials\") pod \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " Nov 03 20:43:39.982166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982133 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume\") pod \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " Nov 03 20:43:39.982243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982230 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm\") pod \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\" (UID: \"6b69c13d-cea1-4e05-a9fa-c22dcc07f852\") " Nov 03 20:43:39.982377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982353 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume" pod "6b69c13d-cea1-4e05-a9fa-c22dcc07f852" (UID: "6b69c13d-cea1-4e05-a9fa-c22dcc07f852"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:39.982553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982528 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume" pod "6b69c13d-cea1-4e05-a9fa-c22dcc07f852" (UID: "6b69c13d-cea1-4e05-a9fa-c22dcc07f852"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:39.982971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982908 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:39.982971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.982930 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:39.984402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.984378 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials" pod "6b69c13d-cea1-4e05-a9fa-c22dcc07f852" (UID: "6b69c13d-cea1-4e05-a9fa-c22dcc07f852"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:39.984484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:39.984415 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm" pod "6b69c13d-cea1-4e05-a9fa-c22dcc07f852" (UID: "6b69c13d-cea1-4e05-a9fa-c22dcc07f852"). InnerVolumeSpecName "kube-api-access-f2bcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:40.084617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.084581 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:40.084617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.084612 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2bcm\" (UniqueName: \"kubernetes.io/projected/6b69c13d-cea1-4e05-a9fa-c22dcc07f852-kube-api-access-f2bcm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:40.806973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.806945 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" Nov 03 20:43:40.807472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.806980 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-598bd85469-hbkd8" event={"ID":"6b69c13d-cea1-4e05-a9fa-c22dcc07f852","Type":"ContainerDied","Data":"1fb14d1c4bf3cc859250f3b054520cc31fa14470e637c8154de4f875bbb32898"} Nov 03 20:43:40.807472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.807044 5815 scope.go:122] "RemoveContainer" containerID="67e5293f62df685277d729b4c54a476c4e8c267be1cce953f8aaf93a2016ad92" Nov 03 20:43:40.837242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.837212 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-598bd85469-hbkd8"] Nov 03 20:43:40.847528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:40.847442 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-598bd85469-hbkd8"] Nov 03 20:43:42.159761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.159727 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm"] Nov 03 20:43:42.161673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.161647 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6b69c13d-cea1-4e05-a9fa-c22dcc07f852" containerName="payload-processing" Nov 03 20:43:42.161673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.161671 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b69c13d-cea1-4e05-a9fa-c22dcc07f852" containerName="payload-processing" Nov 03 20:43:42.162083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.162064 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b69c13d-cea1-4e05-a9fa-c22dcc07f852" containerName="payload-processing" Nov 03 20:43:42.193124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.193091 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm"] Nov 03 20:43:42.193457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.193426 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.200553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.200480 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d4fe2\"" Nov 03 20:43:42.201851 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.201635 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj"] Nov 03 20:43:42.212551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.210749 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj"] Nov 03 20:43:42.212551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.210841 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.216617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.215407 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3d4fe2-dockercfg-ks9s8\"" Nov 03 20:43:42.264330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.264302 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twnz5\" (UniqueName: \"kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5\") pod \"payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.264474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.264448 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.264592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.264566 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.265564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.264830 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzpx\" (UniqueName: \"kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx\") pod \"payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.389528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.374753 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twnz5\" (UniqueName: \"kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5\") pod \"payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.389528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.374874 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.389528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.374937 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.389528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.375230 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzpx\" (UniqueName: \"kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx\") pod \"payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.389528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.376623 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.409418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.402583 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.427574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.415191 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twnz5\" (UniqueName: \"kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5\") pod \"payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.427574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.418296 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzpx\" (UniqueName: \"kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx\") pod \"payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.525564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.521128 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:42.538532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.532857 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:42.689533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:42.689419 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b69c13d-cea1-4e05-a9fa-c22dcc07f852" path="/var/lib/kubelet/pods/6b69c13d-cea1-4e05-a9fa-c22dcc07f852/volumes" Nov 03 20:43:43.024341 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:43:43.023291 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcf54e9_a39e_4d5d_b081_0af49229d668.slice/crio-010e597a0d79360ae0fb4c5d43d7ddb817f5de36d80664697bb811ebe16e71d1 WatchSource:0}: Error finding container 010e597a0d79360ae0fb4c5d43d7ddb817f5de36d80664697bb811ebe16e71d1: Status 404 returned error can't find the container with id 010e597a0d79360ae0fb4c5d43d7ddb817f5de36d80664697bb811ebe16e71d1 Nov 03 20:43:43.025959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.025932 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm"] Nov 03 20:43:43.026904 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:43:43.026879 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb738ce5e_b7ca_45ff_a720_1ff918fa2ee6.slice/crio-6177cfb502763535cae6e46cabe89742715be38055a2d1699e9f5694fdbf7966 WatchSource:0}: Error finding container 6177cfb502763535cae6e46cabe89742715be38055a2d1699e9f5694fdbf7966: Status 404 returned error can't find the container with id 6177cfb502763535cae6e46cabe89742715be38055a2d1699e9f5694fdbf7966 Nov 03 20:43:43.032367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.032348 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj"] Nov 03 20:43:43.844755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.844702 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" event={"ID":"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6","Type":"ContainerStarted","Data":"a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f"} Nov 03 20:43:43.845252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.844772 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:43.845252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.844799 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" event={"ID":"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6","Type":"ContainerStarted","Data":"6177cfb502763535cae6e46cabe89742715be38055a2d1699e9f5694fdbf7966"} Nov 03 20:43:43.847129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.847096 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" event={"ID":"cbcf54e9-a39e-4d5d-b081-0af49229d668","Type":"ContainerStarted","Data":"225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966"} Nov 03 20:43:43.847295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.847149 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" event={"ID":"cbcf54e9-a39e-4d5d-b081-0af49229d668","Type":"ContainerStarted","Data":"010e597a0d79360ae0fb4c5d43d7ddb817f5de36d80664697bb811ebe16e71d1"} Nov 03 20:43:43.847295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.847176 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:43.855071 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:43:43.855041 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:43:43.865555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.865486 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" podStartSLOduration=1.865470992 podStartE2EDuration="1.865470992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:43.863158373 +0000 UTC m=+2533.393504048" watchObservedRunningTime="2026-11-03 20:43:43.865470992 +0000 UTC m=+2533.395816664" Nov 03 20:43:43.885876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:43.885829 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" podStartSLOduration=1.885810953 podStartE2EDuration="1.885810953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:43.88040579 +0000 UTC m=+2533.410751463" watchObservedRunningTime="2026-11-03 20:43:43.885810953 +0000 UTC m=+2533.416156625" Nov 03 20:43:44.696679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:44.696647 5815 scope.go:122] "RemoveContainer" containerID="21d3e2413184e1de9cda455eb9ba09c9e638a3a3b8658cbd21342faa959cb3ba" Nov 03 20:43:44.717986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:44.717955 5815 scope.go:122] "RemoveContainer" containerID="93beb12fc98f8f0414e524fcd25c54446078858068cdbfdad7207f13d7b61ef7" Nov 03 20:43:51.497308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.497273 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q"] Nov 03 20:43:51.506473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.506446 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.523954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.523922 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c6934-openshift-default-dockercfg-jgdtj\"" Nov 03 20:43:51.536844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.536804 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q"] Nov 03 20:43:51.563928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.563886 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564006 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564094 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564326 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564470 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hv8\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564576 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564605 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564655 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.564741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.564695 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.688986 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689218 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hv8\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689279 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689310 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689390 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689462 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689638 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689697 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.690532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.689753 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.724590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.719019 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.724590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.719258 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.724590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.719866 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.724590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.720053 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.724590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.720461 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.724590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.720824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.736991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.727121 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hv8\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.765208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.764855 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.783706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.782079 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:51.861530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:51.860883 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:52.435018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:52.434952 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q"] Nov 03 20:43:52.435466 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:43:52.435411 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd144df_a006_4b38_8d0e_dc3295210007.slice/crio-5b8a83efd42d69edc2298349ffd982f77c2fafcb99034801b462e32befc538b1 WatchSource:0}: Error finding container 5b8a83efd42d69edc2298349ffd982f77c2fafcb99034801b462e32befc538b1: Status 404 returned error can't find the container with id 5b8a83efd42d69edc2298349ffd982f77c2fafcb99034801b462e32befc538b1 Nov 03 20:43:52.438411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:52.438377 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:43:52.438495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:52.438449 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:43:52.438495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:52.438481 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:43:53.091837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:53.091786 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" event={"ID":"bbd144df-a006-4b38-8d0e-dc3295210007","Type":"ContainerStarted","Data":"b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7"} Nov 03 20:43:53.092363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:53.091863 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" event={"ID":"bbd144df-a006-4b38-8d0e-dc3295210007","Type":"ContainerStarted","Data":"5b8a83efd42d69edc2298349ffd982f77c2fafcb99034801b462e32befc538b1"} Nov 03 20:43:53.127556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:53.125669 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podStartSLOduration=2.125646373 podStartE2EDuration="2.125646373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:53.115905329 +0000 UTC m=+2542.646251003" watchObservedRunningTime="2026-11-03 20:43:53.125646373 +0000 UTC m=+2542.655992069" Nov 03 20:43:53.861965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:53.861935 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:53.868726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:53.868698 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:54.102886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:54.102856 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:54.103969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:54.103944 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:43:54.862645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:54.862615 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:43:54.862831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:54.862696 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:43:57.988105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:43:57.988060 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:44:12.062542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:12.059592 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69"] Nov 03 20:44:12.080963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:12.080926 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776f3f71-jws69"] Nov 03 20:44:12.588966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:12.588929 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24410932-5164-4c55-adf9-dd37015fe294" path="/var/lib/kubelet/pods/24410932-5164-4c55-adf9-dd37015fe294/volumes" Nov 03 20:44:13.853369 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:44:13.853335 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:44:27.510779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.510746 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw"] Nov 03 20:44:27.516217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.516193 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.522169 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.522127 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c94fdf-openshift-default-dockercfg-72cm7\"" Nov 03 20:44:27.526939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.526918 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw"] Nov 03 20:44:27.689592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.689557 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.698985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.698937 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.711555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.699231 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.719756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.719717 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tff72\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.720053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.720024 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.720291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.720273 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.720462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.720447 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.720838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.720816 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.721142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.721121 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.822727 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.822847 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.822914 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.823018 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.823185 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.823277 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.823305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.823330 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.823407 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tff72\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.824262 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.825048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.824718 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.826839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.826814 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.828065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.828045 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.828560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.828534 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.862544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.860002 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.884549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.868894 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.884549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.869642 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tff72\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:27.898542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:27.886709 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo\") pod \"e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:28.159773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:28.154945 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:28.546038 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:44:28.546008 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c95d6d4_ccfb_4178_a3cb_72700e6de96d.slice/crio-ee335a35b001b08225640fc54fbc64d252562ccaa5d28b2440cff1bd0d19f680 WatchSource:0}: Error finding container ee335a35b001b08225640fc54fbc64d252562ccaa5d28b2440cff1bd0d19f680: Status 404 returned error can't find the container with id ee335a35b001b08225640fc54fbc64d252562ccaa5d28b2440cff1bd0d19f680 Nov 03 20:44:28.547333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:28.547297 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw"] Nov 03 20:44:28.549392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:28.549356 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:28.549479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:28.549448 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:28.549537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:28.549481 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:29.537743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:29.537705 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" event={"ID":"0c95d6d4-ccfb-4178-a3cb-72700e6de96d","Type":"ContainerStarted","Data":"4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1"} Nov 03 20:44:29.537966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:29.537773 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" event={"ID":"0c95d6d4-ccfb-4178-a3cb-72700e6de96d","Type":"ContainerStarted","Data":"ee335a35b001b08225640fc54fbc64d252562ccaa5d28b2440cff1bd0d19f680"} Nov 03 20:44:29.565412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:29.565360 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podStartSLOduration=2.56534611 podStartE2EDuration="2.56534611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:29.563212006 +0000 UTC m=+2579.093557677" watchObservedRunningTime="2026-11-03 20:44:29.56534611 +0000 UTC m=+2579.095691781" Nov 03 20:44:30.155998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:30.155962 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:30.162798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:30.162762 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:30.547366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:30.547336 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:30.548403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:30.548379 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:44:33.278454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:33.278421 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:44:43.878103 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:44:43.878028 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:44:44.792555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:44.789718 5815 scope.go:122] "RemoveContainer" containerID="7d9904cefbe2ff836987486cbc7274bf95c568e222ec571cb5134172b8866f3d" Nov 03 20:44:44.862869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:44.861989 5815 scope.go:122] "RemoveContainer" containerID="1b9ae43530da21a854bde59db1c9faf369a1986bd4579b0010f77dbf36236360" Nov 03 20:44:44.904781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:44.904639 5815 scope.go:122] "RemoveContainer" containerID="0d6b825e55fc59190c71903d2b34129f3bacbdef3e8a76f530bf114e4bbb8559" Nov 03 20:44:48.229261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:48.229228 5815 generic.go:356] "Generic (PLEG): container finished" podID="bbd144df-a006-4b38-8d0e-dc3295210007" containerID="b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7" exitCode=0 Nov 03 20:44:48.229732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:48.229309 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" event={"ID":"bbd144df-a006-4b38-8d0e-dc3295210007","Type":"ContainerDied","Data":"b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7"} Nov 03 20:44:48.230007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:48.229986 5815 scope.go:122] "RemoveContainer" containerID="b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7" Nov 03 20:44:48.232089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:48.232052 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:48.232234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:48.232142 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:48.232234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:48.232195 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:49.240056 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.240018 5815 generic.go:356] "Generic (PLEG): container finished" podID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerID="02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1" exitCode=0 Nov 03 20:44:49.240647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.240071 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" event={"ID":"b6a28dc7-b27e-4c27-97ee-38996ee945b5","Type":"ContainerDied","Data":"02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1"} Nov 03 20:44:49.240647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.240603 5815 scope.go:122] "RemoveContainer" containerID="02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1" Nov 03 20:44:49.242580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.242559 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" event={"ID":"bbd144df-a006-4b38-8d0e-dc3295210007","Type":"ContainerStarted","Data":"5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6"} Nov 03 20:44:49.242857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.242826 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:49.242914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.242899 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:49.242952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.242931 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:44:49.862021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.861993 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:44:49.862857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.862837 5815 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:44:49.862958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:49.862883 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:44:50.253562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:50.253492 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" event={"ID":"b6a28dc7-b27e-4c27-97ee-38996ee945b5","Type":"ContainerStarted","Data":"3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057"} Nov 03 20:44:50.374535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:50.372289 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:44:50.391459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:50.376939 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:50.391459 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:50.376992 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:50.861697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:50.861660 5815 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:44:50.861984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:50.861719 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:44:51.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:51.389623 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:51.416532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:51.389681 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:51.861646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:51.861609 5815 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:44:51.861935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:51.861903 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:44:52.372933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:52.372896 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:52.378626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:52.378580 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:52.862427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:52.861927 5815 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:44:52.862427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:52.861994 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:44:53.369532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:53.367148 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:53.369532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:53.367209 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:53.869369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:53.868280 5815 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:44:53.869369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:53.868357 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:44:54.108703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:54.107605 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:44:54.371530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:54.370890 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:54.371530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:54.370950 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:54.793664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:54.793625 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:44:54.861923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:54.861887 5815 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" start-of-body= Nov 03 20:44:54.862100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:54.861945 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.90:15021/healthz/ready\": dial tcp 10.132.1.90:15021: connect: connection refused" Nov 03 20:44:55.380451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:55.368915 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:55.380451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:55.368986 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:56.082843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:56.082808 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:44:56.088250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:56.088222 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:44:56.384325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:56.383825 5815 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" start-of-body= Nov 03 20:44:56.384325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:56.383878 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.87:15021/healthz/ready\": dial tcp 10.132.1.87:15021: connect: connection refused" Nov 03 20:44:57.028727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:57.028684 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh"] Nov 03 20:44:57.034027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:57.033996 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089907d9-jvvnh"] Nov 03 20:44:57.373136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:57.373058 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:44:57.374154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:57.374125 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:44:58.605302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:44:58.605268 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab487ad-1ccf-4a5e-80e8-e480f3273908" path="/var/lib/kubelet/pods/4ab487ad-1ccf-4a5e-80e8-e480f3273908/volumes" Nov 03 20:45:00.172518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.172467 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf"] Nov 03 20:45:00.195035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.194999 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.206626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.206476 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8"] Nov 03 20:45:00.211975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.211936 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:45:00.216735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.216713 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.222762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.221885 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8"] Nov 03 20:45:00.232031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.232013 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.237834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.237813 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq"] Nov 03 20:45:00.245786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.244263 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.254913 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.254891 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z"] Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276226 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276295 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjxc\" (UniqueName: \"kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276564 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276600 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh2sd\" (UniqueName: \"kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd\") pod \"maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276712 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276755 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46lwx\" (UniqueName: \"kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx\") pod \"maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276801 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.278592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.276830 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hpv\" (UniqueName: \"kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.284457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.284418 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf"] Nov 03 20:45:00.284457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.284461 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8"] Nov 03 20:45:00.284680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.284598 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.284912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.284895 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t"] Nov 03 20:45:00.313804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.313784 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8"] Nov 03 20:45:00.314102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.313982 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.314102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.314021 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq"] Nov 03 20:45:00.318014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.317407 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g"] Nov 03 20:45:00.328060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.327928 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.336206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.333813 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf"] Nov 03 20:45:00.341648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.341629 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm"] Nov 03 20:45:00.341865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.341850 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.348059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.348039 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5"] Nov 03 20:45:00.348724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.348707 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.379768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.379744 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.379940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.379925 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.380043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380032 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh2sd\" (UniqueName: \"kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd\") pod \"maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.380127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380116 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6jw\" (UniqueName: \"kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.380241 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380230 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.380314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380305 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5h5r\" (UniqueName: \"kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.380422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380412 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46lwx\" (UniqueName: \"kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx\") pod \"maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.380722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380499 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.380802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380767 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hpv\" (UniqueName: \"kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.380946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.380925 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pfm\" (UniqueName: \"kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.381019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.381004 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.381206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.381183 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.381689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.381662 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.381914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.381838 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.382176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.382128 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.382353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.382327 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.382663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.382544 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.383131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.383096 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvl27\" (UniqueName: \"kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.383197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.383164 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.383237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.383196 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.383274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.383242 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjxc\" (UniqueName: \"kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.384198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.384183 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.384855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.384838 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z"] Nov 03 20:45:00.385027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.385017 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.390747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.390713 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6"] Nov 03 20:45:00.399833 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.399809 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46lwx\" (UniqueName: \"kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx\") pod \"maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.401541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.400933 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh2sd\" (UniqueName: \"kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd\") pod \"maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.403065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.403040 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjxc\" (UniqueName: \"kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.403728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.403706 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl"] Nov 03 20:45:00.404327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.404306 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.411286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.411272 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh"] Nov 03 20:45:00.411481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.411462 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.418950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.418922 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hpv\" (UniqueName: \"kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.419376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.419354 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq"] Nov 03 20:45:00.419868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.419446 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.440032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.439717 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf"] Nov 03 20:45:00.440032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.439760 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g"] Nov 03 20:45:00.440032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.439840 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.447947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.447927 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5"] Nov 03 20:45:00.457485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.456225 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l"] Nov 03 20:45:00.465625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.465604 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh"] Nov 03 20:45:00.465802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.465641 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.472106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.472088 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx"] Nov 03 20:45:00.479288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.479268 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.481431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.481415 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l"] Nov 03 20:45:00.484789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484431 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwck\" (UniqueName: \"kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.484789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484493 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pfm\" (UniqueName: \"kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.484789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484543 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thkql\" (UniqueName: \"kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.484789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484570 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.484789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484607 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.484789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484650 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484815 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484851 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484926 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484965 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5sq\" (UniqueName: \"kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.484994 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvl27\" (UniqueName: \"kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485033 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485059 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.485207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485137 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.485807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485289 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.485807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485374 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6jw\" (UniqueName: \"kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.485807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485410 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwsp\" (UniqueName: \"kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.485807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485478 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5h5r\" (UniqueName: \"kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.486914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.485524 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wr4\" (UniqueName: \"kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.486914 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.486033 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.489878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.489349 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.489878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.489548 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk"] Nov 03 20:45:00.491294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.490642 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.491294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.491237 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.492731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.491955 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.492731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.492717 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.502718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.502578 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pfm\" (UniqueName: \"kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.504952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.504907 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6jw\" (UniqueName: \"kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.506866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.506843 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvl27\" (UniqueName: \"kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.510192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.510170 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.511488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.511459 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5h5r\" (UniqueName: \"kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.511695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.511565 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k"] Nov 03 20:45:00.527242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.527181 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.544452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.544422 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk"] Nov 03 20:45:00.544676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.544470 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t"] Nov 03 20:45:00.544676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.544595 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.550639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.550616 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k"] Nov 03 20:45:00.557007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.556983 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6"] Nov 03 20:45:00.561060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.561040 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl"] Nov 03 20:45:00.566228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.566208 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx"] Nov 03 20:45:00.571340 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.571317 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm"] Nov 03 20:45:00.587007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.586986 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:00.587852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.587828 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wr4\" (UniqueName: \"kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.587976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.587880 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.587976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.587940 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwck\" (UniqueName: \"kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.587976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.587970 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xpts\" (UniqueName: \"kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.588189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588021 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thkql\" (UniqueName: \"kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.588189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588045 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.588189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588082 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.588189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588111 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.588669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588640 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626hn\" (UniqueName: \"kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.588784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588721 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.588855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588814 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5sq\" (UniqueName: \"kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.588936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.588919 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.589022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589001 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.589164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589139 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwsp\" (UniqueName: \"kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.589235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589159 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.589235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589210 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.590206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589249 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.590206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589750 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.590206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.589851 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.603325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.603256 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5sq\" (UniqueName: \"kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.603658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.603636 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thkql\" (UniqueName: \"kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:00.605400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.605353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wr4\" (UniqueName: \"kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.606231 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.606208 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwck\" (UniqueName: \"kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.616659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.616622 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq"] Nov 03 20:45:00.622624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.622600 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:00.623471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.623448 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwsp\" (UniqueName: \"kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.631161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.631119 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:00.640278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.640253 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:00.660020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.659965 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:00.679571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.678977 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:00.687143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.687115 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:00.696034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.695839 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:00.709302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.709268 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626hn\" (UniqueName: \"kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.709560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.709536 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9xd\" (UniqueName: \"kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd\") pod \"maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.709675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.709587 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.709766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.709702 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.709876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.709828 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.709876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.709862 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9h88\" (UniqueName: \"kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.717073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.710014 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xpts\" (UniqueName: \"kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.717073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.710114 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.717073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.713894 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.717073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.714196 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:00.717073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.714794 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.752128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.743266 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xpts\" (UniqueName: \"kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.752128 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.749891 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626hn\" (UniqueName: \"kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.765749 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.765538 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:00.799134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.798997 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:00.803142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.802575 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:00.820875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.817616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9xd\" (UniqueName: \"kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd\") pod \"maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.820875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.817673 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.820875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.817769 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.820875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.817863 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9h88\" (UniqueName: \"kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.820875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.819499 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.820875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.820204 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.850146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.850028 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9h88\" (UniqueName: \"kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:00.865225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.864958 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:00.885798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.878894 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:00.892890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.892820 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9xd\" (UniqueName: \"kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd\") pod \"maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:00.896017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.895953 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:00.904575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:00.904523 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:01.016191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.015923 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:01.038431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.026348 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:01.353366 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.353312 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f86469_cd81_4245_87c2_c487b3fc21f0.slice/crio-40b8ac70027cef144f01e0dae5593b1006c91c249e94f0c3742ffb28c0f63f27 WatchSource:0}: Error finding container 40b8ac70027cef144f01e0dae5593b1006c91c249e94f0c3742ffb28c0f63f27: Status 404 returned error can't find the container with id 40b8ac70027cef144f01e0dae5593b1006c91c249e94f0c3742ffb28c0f63f27 Nov 03 20:45:01.355841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.355817 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8"] Nov 03 20:45:01.384473 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.384308 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd97fec_515e_4077_98ce_3257ff4316e1.slice/crio-392d550495213b3f1faa2779187969234c6d66cd181818d20b8ca761d678087d WatchSource:0}: Error finding container 392d550495213b3f1faa2779187969234c6d66cd181818d20b8ca761d678087d: Status 404 returned error can't find the container with id 392d550495213b3f1faa2779187969234c6d66cd181818d20b8ca761d678087d Nov 03 20:45:01.405536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.405494 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf"] Nov 03 20:45:01.430720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.430693 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z"] Nov 03 20:45:01.651586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.651548 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerStarted","Data":"40b8ac70027cef144f01e0dae5593b1006c91c249e94f0c3742ffb28c0f63f27"} Nov 03 20:45:01.653284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.653258 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" event={"ID":"9bd97fec-515e-4077-98ce-3257ff4316e1","Type":"ContainerStarted","Data":"392d550495213b3f1faa2779187969234c6d66cd181818d20b8ca761d678087d"} Nov 03 20:45:01.654943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.654921 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerStarted","Data":"66b66a28ae73b5e1544c555a8da6f92e8d3545d8e46c4f7db790a73756b4837b"} Nov 03 20:45:01.699309 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.698577 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea08089a_857b_4432_8549_50ca4bf1dbbc.slice/crio-1b12b62895018606cd4d1af2b836c798b9c94404608ed7b209f752da61167b68 WatchSource:0}: Error finding container 1b12b62895018606cd4d1af2b836c798b9c94404608ed7b209f752da61167b68: Status 404 returned error can't find the container with id 1b12b62895018606cd4d1af2b836c798b9c94404608ed7b209f752da61167b68 Nov 03 20:45:01.700345 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.700300 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54df193_ba22_440a_ad8c_28f8a9a93f32.slice/crio-1c692e24145b9173d0cd33dd38aa8eea163565a4aef048b9ba13bc473dea88bc WatchSource:0}: Error finding container 1c692e24145b9173d0cd33dd38aa8eea163565a4aef048b9ba13bc473dea88bc: Status 404 returned error can't find the container with id 1c692e24145b9173d0cd33dd38aa8eea163565a4aef048b9ba13bc473dea88bc Nov 03 20:45:01.706878 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.706827 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g"] Nov 03 20:45:01.719670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.719394 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l"] Nov 03 20:45:01.727393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.727366 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t"] Nov 03 20:45:01.736805 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.736768 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8"] Nov 03 20:45:01.802605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.802567 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq"] Nov 03 20:45:01.887722 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.879095 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9666a784_df78_4aaa_9bbe_c6204efd4705.slice/crio-cbb4964e23e9f4aeaf5715f314ae25af3cc5ed7bc45c7c9bff341a2c38870fe4 WatchSource:0}: Error finding container cbb4964e23e9f4aeaf5715f314ae25af3cc5ed7bc45c7c9bff341a2c38870fe4: Status 404 returned error can't find the container with id cbb4964e23e9f4aeaf5715f314ae25af3cc5ed7bc45c7c9bff341a2c38870fe4 Nov 03 20:45:01.887722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.887206 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx"] Nov 03 20:45:01.899802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.899633 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk"] Nov 03 20:45:01.908471 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.908442 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15316213_a694_41a6_aad7_adb9b877db9b.slice/crio-2671025f836bd87925a63ebfea9621689e591df3cab9e98a8103e66a94733db7 WatchSource:0}: Error finding container 2671025f836bd87925a63ebfea9621689e591df3cab9e98a8103e66a94733db7: Status 404 returned error can't find the container with id 2671025f836bd87925a63ebfea9621689e591df3cab9e98a8103e66a94733db7 Nov 03 20:45:01.912829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.909160 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf"] Nov 03 20:45:01.919165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.919142 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq"] Nov 03 20:45:01.922436 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:01.922402 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafad8ca6_5d95_4f58_9b90_5cf22546bb9f.slice/crio-ef1ca41b0f432a2e0c01bf337eac919346a2a664e941adb75171f394042768f8 WatchSource:0}: Error finding container ef1ca41b0f432a2e0c01bf337eac919346a2a664e941adb75171f394042768f8: Status 404 returned error can't find the container with id ef1ca41b0f432a2e0c01bf337eac919346a2a664e941adb75171f394042768f8 Nov 03 20:45:01.928277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.925938 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm"] Nov 03 20:45:01.953770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.953741 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k"] Nov 03 20:45:01.970346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.969955 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6"] Nov 03 20:45:01.980040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.979958 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh"] Nov 03 20:45:01.996517 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:01.996483 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl"] Nov 03 20:45:02.009384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.009351 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5"] Nov 03 20:45:02.666682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.666638 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" event={"ID":"9bd97fec-515e-4077-98ce-3257ff4316e1","Type":"ContainerStarted","Data":"bbf5d03745c5e4462e45afa947558898f44053b080cc774c41545637e3c2fc66"} Nov 03 20:45:02.670177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.670153 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerStarted","Data":"cbb4964e23e9f4aeaf5715f314ae25af3cc5ed7bc45c7c9bff341a2c38870fe4"} Nov 03 20:45:02.671807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.671781 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerStarted","Data":"2671025f836bd87925a63ebfea9621689e591df3cab9e98a8103e66a94733db7"} Nov 03 20:45:02.673390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.673366 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" event={"ID":"24929273-c9e0-4a7f-9ad4-4d23cc831231","Type":"ContainerStarted","Data":"1d21d1afebc377c8f7bbb87a2656cd9c75a26f6010b4591a1f4233739978a9d5"} Nov 03 20:45:02.676734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.676711 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerStarted","Data":"b977d5090515138002e4cbaa3fe7cd17c23be26f945b7fb5a6b5cd9800ea7576"} Nov 03 20:45:02.676847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.676754 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerStarted","Data":"ef4c568d9e279629bdbf12004b17391839df95af0c09e1de1e96c1eda4dee4da"} Nov 03 20:45:02.678174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.678148 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerStarted","Data":"cc3171e35ef4626e0d69fef4469b3cc76d1dae12a22ffdcbf1c19e11a38cb43a"} Nov 03 20:45:02.679736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.679711 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" event={"ID":"0264611c-c245-45a0-bef1-db5ae3b23fdf","Type":"ContainerStarted","Data":"06478a71e9bdf7c74295636b6fdbd92c591f5fe2d927710cfe9d0b23897187f4"} Nov 03 20:45:02.682164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.681907 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" event={"ID":"afad8ca6-5d95-4f58-9b90-5cf22546bb9f","Type":"ContainerStarted","Data":"ef1ca41b0f432a2e0c01bf337eac919346a2a664e941adb75171f394042768f8"} Nov 03 20:45:02.683715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.683691 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerStarted","Data":"93a54e1c4781680138a6b0df732f05abd7b5870b4d1073b0f03d9cbbaaaf5a97"} Nov 03 20:45:02.688122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.687782 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerStarted","Data":"427098de8140ed96d7c703c98414aa9783f5fa27bd5373c5e6227aee88fd514c"} Nov 03 20:45:02.689541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.689379 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" event={"ID":"ea08089a-857b-4432-8549-50ca4bf1dbbc","Type":"ContainerStarted","Data":"1b12b62895018606cd4d1af2b836c798b9c94404608ed7b209f752da61167b68"} Nov 03 20:45:02.690870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.690849 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerStarted","Data":"9947f59893993dd6b5c3458ecfb11629eece60bd3c83e1841d2cb6e7d8ed6b90"} Nov 03 20:45:02.694444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.694420 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" event={"ID":"d54df193-ba22-440a-ad8c-28f8a9a93f32","Type":"ContainerStarted","Data":"1c692e24145b9173d0cd33dd38aa8eea163565a4aef048b9ba13bc473dea88bc"} Nov 03 20:45:02.697409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.697388 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerStarted","Data":"48314be9d14fd1065ad43b5dd3b0d7cf01667ba2397fbd5874da43743490298e"} Nov 03 20:45:02.697631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.696425 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" podStartSLOduration=2.696409433 podStartE2EDuration="2.696409433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.68730121 +0000 UTC m=+2612.217646935" watchObservedRunningTime="2026-11-03 20:45:02.696409433 +0000 UTC m=+2612.226755106" Nov 03 20:45:02.698668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.698645 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerStarted","Data":"fe7307ae0e1026e3ca9cc70f4f319c0ebf07edeee69a33e257e2cc316d2ad5ee"} Nov 03 20:45:02.702451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.702426 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerStarted","Data":"111e760c24207dd936c302fd96901eb5fe10c15815a5f1424f16b7aaeafb0733"} Nov 03 20:45:02.704620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.704554 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" event={"ID":"d43955ae-3dcb-4b1b-9c57-ab375721ed29","Type":"ContainerStarted","Data":"c877c7a44cf73c03ed2fe00b3aebbe24ed50473db70ec42926ef5e41c2c52fe7"} Nov 03 20:45:02.706254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.706233 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerStarted","Data":"815f4f2a0a3faddeaff70a7cc933ae41f4dd6ed41995f1906bcbf8db2113c8a0"} Nov 03 20:45:02.712348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.712309 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" podStartSLOduration=2.7122968160000003 podStartE2EDuration="2.712296816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.707971946 +0000 UTC m=+2612.238317610" watchObservedRunningTime="2026-11-03 20:45:02.712296816 +0000 UTC m=+2612.242642487" Nov 03 20:45:02.731336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.731298 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" podStartSLOduration=2.731286671 podStartE2EDuration="2.731286671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.726619791 +0000 UTC m=+2612.256965490" watchObservedRunningTime="2026-11-03 20:45:02.731286671 +0000 UTC m=+2612.261632342" Nov 03 20:45:02.747908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:02.747872 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" podStartSLOduration=2.7478582879999998 podStartE2EDuration="2.747858288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.744330602 +0000 UTC m=+2612.274676462" watchObservedRunningTime="2026-11-03 20:45:02.747858288 +0000 UTC m=+2612.278203962" Nov 03 20:45:03.724344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.724281 5815 generic.go:356] "Generic (PLEG): container finished" podID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerID="b977d5090515138002e4cbaa3fe7cd17c23be26f945b7fb5a6b5cd9800ea7576" exitCode=60 Nov 03 20:45:03.724764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.724375 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerDied","Data":"b977d5090515138002e4cbaa3fe7cd17c23be26f945b7fb5a6b5cd9800ea7576"} Nov 03 20:45:03.724823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.724759 5815 scope.go:122] "RemoveContainer" containerID="b977d5090515138002e4cbaa3fe7cd17c23be26f945b7fb5a6b5cd9800ea7576" Nov 03 20:45:03.729124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.729097 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerStarted","Data":"fdcc065a118052b8f2810c61c8aea768bf3bbe1ec92aa363911e089464ed55b1"} Nov 03 20:45:03.731196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.731169 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" event={"ID":"afad8ca6-5d95-4f58-9b90-5cf22546bb9f","Type":"ContainerStarted","Data":"f7aeb24c12b9da545eee117c666824d423c43dfd82c739ef9c982ff6f067cc8b"} Nov 03 20:45:03.733777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.733159 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerStarted","Data":"4f20fe1833ed81b3c74c8c7470b81875fbaea6fbb060afe3748903fee1113ebf"} Nov 03 20:45:03.735873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.735844 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerStarted","Data":"1ef04f2ff99eda173fdd1cb578529d13c19f142ebcd417695a0257b047ededb0"} Nov 03 20:45:03.738401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.738376 5815 generic.go:356] "Generic (PLEG): container finished" podID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerID="e0c823db351a1edeabdf1f59eba1dfccd132cf7df6c436ee1b86e3a035b8f3de" exitCode=60 Nov 03 20:45:03.738499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.738462 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" event={"ID":"ea08089a-857b-4432-8549-50ca4bf1dbbc","Type":"ContainerDied","Data":"e0c823db351a1edeabdf1f59eba1dfccd132cf7df6c436ee1b86e3a035b8f3de"} Nov 03 20:45:03.738722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.738705 5815 scope.go:122] "RemoveContainer" containerID="e0c823db351a1edeabdf1f59eba1dfccd132cf7df6c436ee1b86e3a035b8f3de" Nov 03 20:45:03.740672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.740539 5815 generic.go:356] "Generic (PLEG): container finished" podID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerID="9947f59893993dd6b5c3458ecfb11629eece60bd3c83e1841d2cb6e7d8ed6b90" exitCode=60 Nov 03 20:45:03.740672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.740629 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerDied","Data":"9947f59893993dd6b5c3458ecfb11629eece60bd3c83e1841d2cb6e7d8ed6b90"} Nov 03 20:45:03.740869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.740849 5815 scope.go:122] "RemoveContainer" containerID="9947f59893993dd6b5c3458ecfb11629eece60bd3c83e1841d2cb6e7d8ed6b90" Nov 03 20:45:03.748390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.748258 5815 generic.go:356] "Generic (PLEG): container finished" podID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerID="c1f96b1fbaee5924cd125ef20950b49388ef34216ecb8088511079ec25d73f30" exitCode=60 Nov 03 20:45:03.748390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.748338 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" event={"ID":"d54df193-ba22-440a-ad8c-28f8a9a93f32","Type":"ContainerDied","Data":"c1f96b1fbaee5924cd125ef20950b49388ef34216ecb8088511079ec25d73f30"} Nov 03 20:45:03.748612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.748588 5815 scope.go:122] "RemoveContainer" containerID="c1f96b1fbaee5924cd125ef20950b49388ef34216ecb8088511079ec25d73f30" Nov 03 20:45:03.750707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.750680 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerStarted","Data":"d5d2eb8ef0b1c4794425503ef2d0218bf5f49b9f709db2be14db4e226c2a376b"} Nov 03 20:45:03.753718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.752783 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerStarted","Data":"70bd132e9c6b37c24ad7d50e1b5d64d76884cda2f31fff279348e98b3bc87c35"} Nov 03 20:45:03.757820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.757699 5815 generic.go:356] "Generic (PLEG): container finished" podID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerID="111e760c24207dd936c302fd96901eb5fe10c15815a5f1424f16b7aaeafb0733" exitCode=60 Nov 03 20:45:03.757820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.757785 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerDied","Data":"111e760c24207dd936c302fd96901eb5fe10c15815a5f1424f16b7aaeafb0733"} Nov 03 20:45:03.758039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.758020 5815 scope.go:122] "RemoveContainer" containerID="111e760c24207dd936c302fd96901eb5fe10c15815a5f1424f16b7aaeafb0733" Nov 03 20:45:03.774236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.774180 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" podStartSLOduration=3.774148858 podStartE2EDuration="3.774148858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.760016212 +0000 UTC m=+2613.290361886" watchObservedRunningTime="2026-11-03 20:45:03.774148858 +0000 UTC m=+2613.304494531" Nov 03 20:45:03.780910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.780880 5815 generic.go:356] "Generic (PLEG): container finished" podID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerID="5caa3b7b0d6a21c0e53444dff13c3ab9c2c67c39525301c0608715d8d5f047b7" exitCode=60 Nov 03 20:45:03.781194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.781170 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerDied","Data":"5caa3b7b0d6a21c0e53444dff13c3ab9c2c67c39525301c0608715d8d5f047b7"} Nov 03 20:45:03.783085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.781612 5815 scope.go:122] "RemoveContainer" containerID="5caa3b7b0d6a21c0e53444dff13c3ab9c2c67c39525301c0608715d8d5f047b7" Nov 03 20:45:03.790363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.790339 5815 generic.go:356] "Generic (PLEG): container finished" podID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerID="bbf5d03745c5e4462e45afa947558898f44053b080cc774c41545637e3c2fc66" exitCode=60 Nov 03 20:45:03.790465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.790424 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" event={"ID":"9bd97fec-515e-4077-98ce-3257ff4316e1","Type":"ContainerDied","Data":"bbf5d03745c5e4462e45afa947558898f44053b080cc774c41545637e3c2fc66"} Nov 03 20:45:03.791212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.790686 5815 scope.go:122] "RemoveContainer" containerID="bbf5d03745c5e4462e45afa947558898f44053b080cc774c41545637e3c2fc66" Nov 03 20:45:03.794928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.793852 5815 generic.go:356] "Generic (PLEG): container finished" podID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerID="220dfcc1a66f5a3a12d0c15b10370af709b092dc2cf8d9342c27158aab56c492" exitCode=60 Nov 03 20:45:03.794928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.793945 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerDied","Data":"220dfcc1a66f5a3a12d0c15b10370af709b092dc2cf8d9342c27158aab56c492"} Nov 03 20:45:03.794928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.794291 5815 scope.go:122] "RemoveContainer" containerID="220dfcc1a66f5a3a12d0c15b10370af709b092dc2cf8d9342c27158aab56c492" Nov 03 20:45:03.797198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.797076 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" podStartSLOduration=3.797059989 podStartE2EDuration="3.797059989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.792032854 +0000 UTC m=+2613.322378526" watchObservedRunningTime="2026-11-03 20:45:03.797059989 +0000 UTC m=+2613.327405661" Nov 03 20:45:03.798176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.797755 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerStarted","Data":"95c4e8cf22273f54ea3a69d262f6e6dcefec5cd8835c752e29ff10c8e1ebc59d"} Nov 03 20:45:03.807787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.807664 5815 generic.go:356] "Generic (PLEG): container finished" podID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerID="b25f4901cb5ded46ceb40ae7a91f5786e1e5fd2b595d8332eda14234faa24568" exitCode=60 Nov 03 20:45:03.807787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.807761 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" event={"ID":"24929273-c9e0-4a7f-9ad4-4d23cc831231","Type":"ContainerDied","Data":"b25f4901cb5ded46ceb40ae7a91f5786e1e5fd2b595d8332eda14234faa24568"} Nov 03 20:45:03.808206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.808186 5815 scope.go:122] "RemoveContainer" containerID="b25f4901cb5ded46ceb40ae7a91f5786e1e5fd2b595d8332eda14234faa24568" Nov 03 20:45:03.837544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.837483 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" podStartSLOduration=3.837471221 podStartE2EDuration="3.837471221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.830275557 +0000 UTC m=+2613.360621236" watchObservedRunningTime="2026-11-03 20:45:03.837471221 +0000 UTC m=+2613.367816892" Nov 03 20:45:03.905637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.905585 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" podStartSLOduration=3.905567658 podStartE2EDuration="3.905567658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.878738842 +0000 UTC m=+2613.409084516" watchObservedRunningTime="2026-11-03 20:45:03.905567658 +0000 UTC m=+2613.435913331" Nov 03 20:45:03.978882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.978756 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" podStartSLOduration=3.978736108 podStartE2EDuration="3.978736108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.964322742 +0000 UTC m=+2613.494668420" watchObservedRunningTime="2026-11-03 20:45:03.978736108 +0000 UTC m=+2613.509081781" Nov 03 20:45:03.988205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:03.988153 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" podStartSLOduration=3.988136793 podStartE2EDuration="3.988136793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.985246493 +0000 UTC m=+2613.515592188" watchObservedRunningTime="2026-11-03 20:45:03.988136793 +0000 UTC m=+2613.518482467" Nov 03 20:45:04.113006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.112938 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" podStartSLOduration=4.112914898 podStartE2EDuration="4.112914898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:04.075005437 +0000 UTC m=+2613.605351109" watchObservedRunningTime="2026-11-03 20:45:04.112914898 +0000 UTC m=+2613.643260574" Nov 03 20:45:04.891486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.891393 5815 generic.go:356] "Generic (PLEG): container finished" podID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerID="1ef04f2ff99eda173fdd1cb578529d13c19f142ebcd417695a0257b047ededb0" exitCode=60 Nov 03 20:45:04.892244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.892215 5815 scope.go:122] "RemoveContainer" containerID="1ef04f2ff99eda173fdd1cb578529d13c19f142ebcd417695a0257b047ededb0" Nov 03 20:45:04.892473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.892439 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerDied","Data":"1ef04f2ff99eda173fdd1cb578529d13c19f142ebcd417695a0257b047ededb0"} Nov 03 20:45:04.897107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.897077 5815 generic.go:356] "Generic (PLEG): container finished" podID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerID="d5d2eb8ef0b1c4794425503ef2d0218bf5f49b9f709db2be14db4e226c2a376b" exitCode=60 Nov 03 20:45:04.897250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.897180 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerDied","Data":"d5d2eb8ef0b1c4794425503ef2d0218bf5f49b9f709db2be14db4e226c2a376b"} Nov 03 20:45:04.897727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.897686 5815 scope.go:122] "RemoveContainer" containerID="d5d2eb8ef0b1c4794425503ef2d0218bf5f49b9f709db2be14db4e226c2a376b" Nov 03 20:45:04.899772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.899746 5815 generic.go:356] "Generic (PLEG): container finished" podID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerID="70bd132e9c6b37c24ad7d50e1b5d64d76884cda2f31fff279348e98b3bc87c35" exitCode=60 Nov 03 20:45:04.900306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.900286 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerDied","Data":"70bd132e9c6b37c24ad7d50e1b5d64d76884cda2f31fff279348e98b3bc87c35"} Nov 03 20:45:04.900737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.900719 5815 scope.go:122] "RemoveContainer" containerID="70bd132e9c6b37c24ad7d50e1b5d64d76884cda2f31fff279348e98b3bc87c35" Nov 03 20:45:04.904156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.904135 5815 generic.go:356] "Generic (PLEG): container finished" podID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerID="897defce3c145d49231515cc7d09aef4f47a6f85e19a68c09e37f0a3feb1c272" exitCode=60 Nov 03 20:45:04.904910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.904336 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" event={"ID":"d43955ae-3dcb-4b1b-9c57-ab375721ed29","Type":"ContainerDied","Data":"897defce3c145d49231515cc7d09aef4f47a6f85e19a68c09e37f0a3feb1c272"} Nov 03 20:45:04.904910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.904679 5815 scope.go:122] "RemoveContainer" containerID="897defce3c145d49231515cc7d09aef4f47a6f85e19a68c09e37f0a3feb1c272" Nov 03 20:45:04.907176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.907159 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerStarted","Data":"30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1"} Nov 03 20:45:04.908919 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.908900 5815 generic.go:356] "Generic (PLEG): container finished" podID="15316213-a694-41a6-aad7-adb9b877db9b" containerID="95c4e8cf22273f54ea3a69d262f6e6dcefec5cd8835c752e29ff10c8e1ebc59d" exitCode=60 Nov 03 20:45:04.909059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.909037 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerDied","Data":"95c4e8cf22273f54ea3a69d262f6e6dcefec5cd8835c752e29ff10c8e1ebc59d"} Nov 03 20:45:04.909290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.909270 5815 scope.go:122] "RemoveContainer" containerID="95c4e8cf22273f54ea3a69d262f6e6dcefec5cd8835c752e29ff10c8e1ebc59d" Nov 03 20:45:04.911008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.910990 5815 generic.go:356] "Generic (PLEG): container finished" podID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerID="fdcc065a118052b8f2810c61c8aea768bf3bbe1ec92aa363911e089464ed55b1" exitCode=60 Nov 03 20:45:04.911099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.911048 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerDied","Data":"fdcc065a118052b8f2810c61c8aea768bf3bbe1ec92aa363911e089464ed55b1"} Nov 03 20:45:04.911282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.911268 5815 scope.go:122] "RemoveContainer" containerID="fdcc065a118052b8f2810c61c8aea768bf3bbe1ec92aa363911e089464ed55b1" Nov 03 20:45:04.912937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.912918 5815 generic.go:356] "Generic (PLEG): container finished" podID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerID="d9a236d874f592ef37ceabcfc903e6f3d116fe7d79bee4af3d6401aa015f6205" exitCode=60 Nov 03 20:45:04.914232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.914204 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" event={"ID":"0264611c-c245-45a0-bef1-db5ae3b23fdf","Type":"ContainerDied","Data":"d9a236d874f592ef37ceabcfc903e6f3d116fe7d79bee4af3d6401aa015f6205"} Nov 03 20:45:04.914328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.914269 5815 scope.go:122] "RemoveContainer" containerID="d9a236d874f592ef37ceabcfc903e6f3d116fe7d79bee4af3d6401aa015f6205" Nov 03 20:45:04.918285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.918262 5815 generic.go:356] "Generic (PLEG): container finished" podID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerID="f7aeb24c12b9da545eee117c666824d423c43dfd82c739ef9c982ff6f067cc8b" exitCode=60 Nov 03 20:45:04.918393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.918333 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" event={"ID":"afad8ca6-5d95-4f58-9b90-5cf22546bb9f","Type":"ContainerDied","Data":"f7aeb24c12b9da545eee117c666824d423c43dfd82c739ef9c982ff6f067cc8b"} Nov 03 20:45:04.919820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.919800 5815 scope.go:122] "RemoveContainer" containerID="f7aeb24c12b9da545eee117c666824d423c43dfd82c739ef9c982ff6f067cc8b" Nov 03 20:45:04.920578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.919982 5815 generic.go:356] "Generic (PLEG): container finished" podID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerID="4f20fe1833ed81b3c74c8c7470b81875fbaea6fbb060afe3748903fee1113ebf" exitCode=60 Nov 03 20:45:04.920578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.920025 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerDied","Data":"4f20fe1833ed81b3c74c8c7470b81875fbaea6fbb060afe3748903fee1113ebf"} Nov 03 20:45:04.920578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.920338 5815 scope.go:122] "RemoveContainer" containerID="4f20fe1833ed81b3c74c8c7470b81875fbaea6fbb060afe3748903fee1113ebf" Nov 03 20:45:04.951091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:04.951044 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" podStartSLOduration=4.951027674 podStartE2EDuration="4.951027674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:04.950642265 +0000 UTC m=+2614.480987937" watchObservedRunningTime="2026-11-03 20:45:04.951027674 +0000 UTC m=+2614.481373347" Nov 03 20:45:05.932384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.932313 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerStarted","Data":"bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea"} Nov 03 20:45:05.934609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.934579 5815 generic.go:356] "Generic (PLEG): container finished" podID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerID="5f8ffeb1269029e9c646c3a2b7bb000b2e65a85e2628ba170a3ab651ed0044f4" exitCode=60 Nov 03 20:45:05.934714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.934655 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" event={"ID":"9bd97fec-515e-4077-98ce-3257ff4316e1","Type":"ContainerDied","Data":"5f8ffeb1269029e9c646c3a2b7bb000b2e65a85e2628ba170a3ab651ed0044f4"} Nov 03 20:45:05.934798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.934728 5815 scope.go:122] "RemoveContainer" containerID="bbf5d03745c5e4462e45afa947558898f44053b080cc774c41545637e3c2fc66" Nov 03 20:45:05.935060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.935039 5815 scope.go:122] "RemoveContainer" containerID="5f8ffeb1269029e9c646c3a2b7bb000b2e65a85e2628ba170a3ab651ed0044f4" Nov 03 20:45:05.935548 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.935480 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z_odh-ai-gateway-infra(9bd97fec-515e-4077-98ce-3257ff4316e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" Nov 03 20:45:05.936872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.936851 5815 generic.go:356] "Generic (PLEG): container finished" podID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerID="30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1" exitCode=60 Nov 03 20:45:05.936981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.936935 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerDied","Data":"30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1"} Nov 03 20:45:05.937308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.937290 5815 scope.go:122] "RemoveContainer" containerID="30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1" Nov 03 20:45:05.937744 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.937717 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8_odh-ai-gateway-infra(9666a784-df78-4aaa-9bbe-c6204efd4705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" Nov 03 20:45:05.939260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.939240 5815 generic.go:356] "Generic (PLEG): container finished" podID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerID="a163570ea19a6f6919772e0e0508ac9f7d85de54606c5ea5d57685b49da90e08" exitCode=60 Nov 03 20:45:05.939403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.939342 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" event={"ID":"24929273-c9e0-4a7f-9ad4-4d23cc831231","Type":"ContainerDied","Data":"a163570ea19a6f6919772e0e0508ac9f7d85de54606c5ea5d57685b49da90e08"} Nov 03 20:45:05.939677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.939645 5815 scope.go:122] "RemoveContainer" containerID="a163570ea19a6f6919772e0e0508ac9f7d85de54606c5ea5d57685b49da90e08" Nov 03 20:45:05.940004 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.939984 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t_odh-ai-gateway-infra(24929273-c9e0-4a7f-9ad4-4d23cc831231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" Nov 03 20:45:05.941935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.941916 5815 generic.go:356] "Generic (PLEG): container finished" podID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" exitCode=60 Nov 03 20:45:05.942183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.942165 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerDied","Data":"61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea"} Nov 03 20:45:05.942334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.942321 5815 scope.go:122] "RemoveContainer" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" Nov 03 20:45:05.942731 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.942699 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm_odh-ai-gateway-infra(b508ca37-a709-402b-b160-d9f71c5e1aa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" Nov 03 20:45:05.944551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.944528 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerStarted","Data":"409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252"} Nov 03 20:45:05.947444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.946903 5815 generic.go:356] "Generic (PLEG): container finished" podID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerID="24050d47e08c5b755d05072410578a870563b7b69e036be288588e2011a0340b" exitCode=60 Nov 03 20:45:05.947444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.946997 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" event={"ID":"ea08089a-857b-4432-8549-50ca4bf1dbbc","Type":"ContainerDied","Data":"24050d47e08c5b755d05072410578a870563b7b69e036be288588e2011a0340b"} Nov 03 20:45:05.947444 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.947248 5815 scope.go:122] "RemoveContainer" containerID="24050d47e08c5b755d05072410578a870563b7b69e036be288588e2011a0340b" Nov 03 20:45:05.947915 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.947888 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g_odh-ai-gateway-infra(ea08089a-857b-4432-8549-50ca4bf1dbbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" Nov 03 20:45:05.949221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.949199 5815 generic.go:356] "Generic (PLEG): container finished" podID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerID="5379f109da6fbd232302761c84956a7183767361b0d94b5c00b96ecef4159517" exitCode=60 Nov 03 20:45:05.949315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.949276 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerDied","Data":"5379f109da6fbd232302761c84956a7183767361b0d94b5c00b96ecef4159517"} Nov 03 20:45:05.949554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.949524 5815 scope.go:122] "RemoveContainer" containerID="5379f109da6fbd232302761c84956a7183767361b0d94b5c00b96ecef4159517" Nov 03 20:45:05.950391 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.950363 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf_odh-ai-gateway-infra(b3eb1e83-75bd-4701-82fc-98d2742904c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.954191 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" podStartSLOduration=5.954179766 podStartE2EDuration="5.954179766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:05.950223377 +0000 UTC m=+2615.480569050" watchObservedRunningTime="2026-11-03 20:45:05.954179766 +0000 UTC m=+2615.484525438" Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.954693 5815 generic.go:356] "Generic (PLEG): container finished" podID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerID="ecd2b80279c7919bd54b36d1aaf69ad6defaa8f39c2249bbf22b209c1d3fe730" exitCode=60 Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.954780 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" event={"ID":"d54df193-ba22-440a-ad8c-28f8a9a93f32","Type":"ContainerDied","Data":"ecd2b80279c7919bd54b36d1aaf69ad6defaa8f39c2249bbf22b209c1d3fe730"} Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.954980 5815 scope.go:122] "RemoveContainer" containerID="ecd2b80279c7919bd54b36d1aaf69ad6defaa8f39c2249bbf22b209c1d3fe730" Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.955250 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l_odh-ai-gateway-infra(d54df193-ba22-440a-ad8c-28f8a9a93f32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.956939 5815 generic.go:356] "Generic (PLEG): container finished" podID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerID="07fa560f39eae4cb692684ab28a79bb5efda399c87dea8f83cdf37aeb8c020d2" exitCode=60 Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.956998 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerDied","Data":"07fa560f39eae4cb692684ab28a79bb5efda399c87dea8f83cdf37aeb8c020d2"} Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:05.957204 5815 scope.go:122] "RemoveContainer" containerID="07fa560f39eae4cb692684ab28a79bb5efda399c87dea8f83cdf37aeb8c020d2" Nov 03 20:45:05.957552 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:05.957440 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8_odh-ai-gateway-infra(d8f86469-cd81-4245-87c2-c487b3fc21f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" Nov 03 20:45:06.043714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.043677 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r"] Nov 03 20:45:06.050199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.050166 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ad09fec-9j99r"] Nov 03 20:45:06.212435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.212409 5815 scope.go:122] "RemoveContainer" containerID="220dfcc1a66f5a3a12d0c15b10370af709b092dc2cf8d9342c27158aab56c492" Nov 03 20:45:06.361532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.337693 5815 scope.go:122] "RemoveContainer" containerID="b25f4901cb5ded46ceb40ae7a91f5786e1e5fd2b595d8332eda14234faa24568" Nov 03 20:45:06.429802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.426402 5815 scope.go:122] "RemoveContainer" containerID="b977d5090515138002e4cbaa3fe7cd17c23be26f945b7fb5a6b5cd9800ea7576" Nov 03 20:45:06.619944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.619734 5815 scope.go:122] "RemoveContainer" containerID="e0c823db351a1edeabdf1f59eba1dfccd132cf7df6c436ee1b86e3a035b8f3de" Nov 03 20:45:06.628429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.628394 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6037f8-099b-4f35-a143-1097f98e08f0" path="/var/lib/kubelet/pods/2f6037f8-099b-4f35-a143-1097f98e08f0/volumes" Nov 03 20:45:06.648442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.648423 5815 scope.go:122] "RemoveContainer" containerID="9947f59893993dd6b5c3458ecfb11629eece60bd3c83e1841d2cb6e7d8ed6b90" Nov 03 20:45:06.663817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.663796 5815 scope.go:122] "RemoveContainer" containerID="c1f96b1fbaee5924cd125ef20950b49388ef34216ecb8088511079ec25d73f30" Nov 03 20:45:06.882619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:06.880822 5815 scope.go:122] "RemoveContainer" containerID="111e760c24207dd936c302fd96901eb5fe10c15815a5f1424f16b7aaeafb0733" Nov 03 20:45:07.091314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.080701 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" event={"ID":"afad8ca6-5d95-4f58-9b90-5cf22546bb9f","Type":"ContainerStarted","Data":"35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8"} Nov 03 20:45:07.123313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.122458 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerStarted","Data":"3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409"} Nov 03 20:45:07.127025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.126094 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerStarted","Data":"4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df"} Nov 03 20:45:07.129772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.129751 5815 scope.go:122] "RemoveContainer" containerID="24050d47e08c5b755d05072410578a870563b7b69e036be288588e2011a0340b" Nov 03 20:45:07.130192 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.130168 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g_odh-ai-gateway-infra(ea08089a-857b-4432-8549-50ca4bf1dbbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" Nov 03 20:45:07.133081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.132529 5815 scope.go:122] "RemoveContainer" containerID="5379f109da6fbd232302761c84956a7183767361b0d94b5c00b96ecef4159517" Nov 03 20:45:07.133081 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.132768 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf_odh-ai-gateway-infra(b3eb1e83-75bd-4701-82fc-98d2742904c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" Nov 03 20:45:07.135683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.135653 5815 scope.go:122] "RemoveContainer" containerID="ecd2b80279c7919bd54b36d1aaf69ad6defaa8f39c2249bbf22b209c1d3fe730" Nov 03 20:45:07.135922 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.135905 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l_odh-ai-gateway-infra(d54df193-ba22-440a-ad8c-28f8a9a93f32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" Nov 03 20:45:07.138242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.138220 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerStarted","Data":"534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d"} Nov 03 20:45:07.140763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.140742 5815 generic.go:356] "Generic (PLEG): container finished" podID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerID="d6eb4c7181cb7b5ea3adab5fc4d40dcf306a2b7d461f9b882fe537d04e6d5fbb" exitCode=60 Nov 03 20:45:07.140888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.140830 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerDied","Data":"d6eb4c7181cb7b5ea3adab5fc4d40dcf306a2b7d461f9b882fe537d04e6d5fbb"} Nov 03 20:45:07.140955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.140893 5815 scope.go:122] "RemoveContainer" containerID="70bd132e9c6b37c24ad7d50e1b5d64d76884cda2f31fff279348e98b3bc87c35" Nov 03 20:45:07.141236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.141216 5815 scope.go:122] "RemoveContainer" containerID="d6eb4c7181cb7b5ea3adab5fc4d40dcf306a2b7d461f9b882fe537d04e6d5fbb" Nov 03 20:45:07.141867 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.141680 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5_odh-ai-gateway-infra(f035d917-9f8f-417e-9951-1bda4a7f6779)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" Nov 03 20:45:07.143832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.143783 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" event={"ID":"d43955ae-3dcb-4b1b-9c57-ab375721ed29","Type":"ContainerStarted","Data":"d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd"} Nov 03 20:45:07.144262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.144090 5815 scope.go:122] "RemoveContainer" containerID="d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd" Nov 03 20:45:07.144622 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.144384 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk_odh-ai-gateway-infra(d43955ae-3dcb-4b1b-9c57-ab375721ed29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" Nov 03 20:45:07.148832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.148709 5815 generic.go:356] "Generic (PLEG): container finished" podID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerID="bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea" exitCode=60 Nov 03 20:45:07.148832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.148745 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerDied","Data":"bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea"} Nov 03 20:45:07.149272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.149158 5815 scope.go:122] "RemoveContainer" containerID="bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea" Nov 03 20:45:07.149580 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.149496 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq_odh-ai-gateway-infra(84ae2e98-dfd4-4a54-9546-c1cdf14c601a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" Nov 03 20:45:07.154547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.151643 5815 scope.go:122] "RemoveContainer" containerID="5f8ffeb1269029e9c646c3a2b7bb000b2e65a85e2628ba170a3ab651ed0044f4" Nov 03 20:45:07.154547 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.152232 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z_odh-ai-gateway-infra(9bd97fec-515e-4077-98ce-3257ff4316e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" Nov 03 20:45:07.173568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.173479 5815 scope.go:122] "RemoveContainer" containerID="30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1" Nov 03 20:45:07.173990 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.173962 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8_odh-ai-gateway-infra(9666a784-df78-4aaa-9bbe-c6204efd4705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" Nov 03 20:45:07.176613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.176589 5815 generic.go:356] "Generic (PLEG): container finished" podID="15316213-a694-41a6-aad7-adb9b877db9b" containerID="f16c0213aeecc84197abe5de99515315d33da951790ac1f87ab135a987cc0e39" exitCode=60 Nov 03 20:45:07.176844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.176827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerDied","Data":"f16c0213aeecc84197abe5de99515315d33da951790ac1f87ab135a987cc0e39"} Nov 03 20:45:07.177794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.177779 5815 scope.go:122] "RemoveContainer" containerID="f16c0213aeecc84197abe5de99515315d33da951790ac1f87ab135a987cc0e39" Nov 03 20:45:07.178453 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.178433 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx_odh-ai-gateway-infra(15316213-a694-41a6-aad7-adb9b877db9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" podUID="15316213-a694-41a6-aad7-adb9b877db9b" Nov 03 20:45:07.180311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.180287 5815 scope.go:122] "RemoveContainer" containerID="a163570ea19a6f6919772e0e0508ac9f7d85de54606c5ea5d57685b49da90e08" Nov 03 20:45:07.180906 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.180880 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t_odh-ai-gateway-infra(24929273-c9e0-4a7f-9ad4-4d23cc831231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" Nov 03 20:45:07.188336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.188313 5815 scope.go:122] "RemoveContainer" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" Nov 03 20:45:07.188754 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.188724 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm_odh-ai-gateway-infra(b508ca37-a709-402b-b160-d9f71c5e1aa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" Nov 03 20:45:07.189768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.189743 5815 generic.go:356] "Generic (PLEG): container finished" podID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" exitCode=60 Nov 03 20:45:07.189883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.189833 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerDied","Data":"409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252"} Nov 03 20:45:07.190064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.190035 5815 scope.go:122] "RemoveContainer" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" Nov 03 20:45:07.190341 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.190318 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh_odh-ai-gateway-infra(c5567729-6f60-4c64-b64d-9f032db9ebe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" Nov 03 20:45:07.192008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.191988 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" event={"ID":"0264611c-c245-45a0-bef1-db5ae3b23fdf","Type":"ContainerStarted","Data":"df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596"} Nov 03 20:45:07.192318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.192300 5815 scope.go:122] "RemoveContainer" containerID="07fa560f39eae4cb692684ab28a79bb5efda399c87dea8f83cdf37aeb8c020d2" Nov 03 20:45:07.192586 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:07.192568 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8_odh-ai-gateway-infra(d8f86469-cd81-4245-87c2-c487b3fc21f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" Nov 03 20:45:07.411842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.411789 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" podStartSLOduration=7.411776285 podStartE2EDuration="7.411776285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:07.407013938 +0000 UTC m=+2616.937359610" watchObservedRunningTime="2026-11-03 20:45:07.411776285 +0000 UTC m=+2616.942121956" Nov 03 20:45:07.518019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.517997 5815 scope.go:122] "RemoveContainer" containerID="5caa3b7b0d6a21c0e53444dff13c3ab9c2c67c39525301c0608715d8d5f047b7" Nov 03 20:45:07.534544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.534520 5815 scope.go:122] "RemoveContainer" containerID="95c4e8cf22273f54ea3a69d262f6e6dcefec5cd8835c752e29ff10c8e1ebc59d" Nov 03 20:45:07.549904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:07.549884 5815 scope.go:122] "RemoveContainer" containerID="fdcc065a118052b8f2810c61c8aea768bf3bbe1ec92aa363911e089464ed55b1" Nov 03 20:45:08.207432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.207387 5815 scope.go:122] "RemoveContainer" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" Nov 03 20:45:08.207974 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.207794 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh_odh-ai-gateway-infra(c5567729-6f60-4c64-b64d-9f032db9ebe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" Nov 03 20:45:08.209713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.209686 5815 generic.go:356] "Generic (PLEG): container finished" podID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerID="df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596" exitCode=60 Nov 03 20:45:08.209862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.209777 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" event={"ID":"0264611c-c245-45a0-bef1-db5ae3b23fdf","Type":"ContainerDied","Data":"df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596"} Nov 03 20:45:08.209862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.209847 5815 scope.go:122] "RemoveContainer" containerID="d9a236d874f592ef37ceabcfc903e6f3d116fe7d79bee4af3d6401aa015f6205" Nov 03 20:45:08.210032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.210016 5815 scope.go:122] "RemoveContainer" containerID="df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596" Nov 03 20:45:08.210319 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.210297 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl_odh-ai-gateway-infra(0264611c-c245-45a0-bef1-db5ae3b23fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" Nov 03 20:45:08.212167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.212145 5815 generic.go:356] "Generic (PLEG): container finished" podID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerID="35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8" exitCode=60 Nov 03 20:45:08.212260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.212233 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" event={"ID":"afad8ca6-5d95-4f58-9b90-5cf22546bb9f","Type":"ContainerDied","Data":"35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8"} Nov 03 20:45:08.212453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.212437 5815 scope.go:122] "RemoveContainer" containerID="35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8" Nov 03 20:45:08.212810 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.212785 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf_odh-ai-gateway-infra(afad8ca6-5d95-4f58-9b90-5cf22546bb9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" Nov 03 20:45:08.214647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.214625 5815 generic.go:356] "Generic (PLEG): container finished" podID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerID="3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409" exitCode=60 Nov 03 20:45:08.214737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.214709 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerDied","Data":"3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409"} Nov 03 20:45:08.214928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.214912 5815 scope.go:122] "RemoveContainer" containerID="3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409" Nov 03 20:45:08.215212 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.215192 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq_odh-ai-gateway-infra(1492ceb7-ca41-4a12-bc1b-d621d4a3118d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" Nov 03 20:45:08.216703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.216683 5815 generic.go:356] "Generic (PLEG): container finished" podID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerID="4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df" exitCode=60 Nov 03 20:45:08.216802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.216780 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerDied","Data":"4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df"} Nov 03 20:45:08.216982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.216964 5815 scope.go:122] "RemoveContainer" containerID="4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df" Nov 03 20:45:08.217285 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.217264 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k_odh-ai-gateway-infra(81040066-6649-49bb-ac78-a1fccda7b2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" Nov 03 20:45:08.218903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.218883 5815 generic.go:356] "Generic (PLEG): container finished" podID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerID="534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d" exitCode=60 Nov 03 20:45:08.219002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.218952 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerDied","Data":"534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d"} Nov 03 20:45:08.219193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.219180 5815 scope.go:122] "RemoveContainer" containerID="534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d" Nov 03 20:45:08.219447 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.219428 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6_odh-ai-gateway-infra(a9feecd9-d5d6-467c-8164-f8f9e6ba7b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" Nov 03 20:45:08.221080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.221062 5815 scope.go:122] "RemoveContainer" containerID="d6eb4c7181cb7b5ea3adab5fc4d40dcf306a2b7d461f9b882fe537d04e6d5fbb" Nov 03 20:45:08.221348 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.221321 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5_odh-ai-gateway-infra(f035d917-9f8f-417e-9951-1bda4a7f6779)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" Nov 03 20:45:08.224481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.224459 5815 generic.go:356] "Generic (PLEG): container finished" podID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerID="d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd" exitCode=60 Nov 03 20:45:08.224614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.224537 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" event={"ID":"d43955ae-3dcb-4b1b-9c57-ab375721ed29","Type":"ContainerDied","Data":"d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd"} Nov 03 20:45:08.224823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.224807 5815 scope.go:122] "RemoveContainer" containerID="d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd" Nov 03 20:45:08.225094 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.225072 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk_odh-ai-gateway-infra(d43955ae-3dcb-4b1b-9c57-ab375721ed29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" Nov 03 20:45:08.226427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.226405 5815 scope.go:122] "RemoveContainer" containerID="bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea" Nov 03 20:45:08.227835 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.227792 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq_odh-ai-gateway-infra(84ae2e98-dfd4-4a54-9546-c1cdf14c601a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" Nov 03 20:45:08.228775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.228707 5815 scope.go:122] "RemoveContainer" containerID="f16c0213aeecc84197abe5de99515315d33da951790ac1f87ab135a987cc0e39" Nov 03 20:45:08.228826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.228797 5815 scope.go:122] "RemoveContainer" containerID="f7aeb24c12b9da545eee117c666824d423c43dfd82c739ef9c982ff6f067cc8b" Nov 03 20:45:08.228943 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:08.228920 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx_odh-ai-gateway-infra(15316213-a694-41a6-aad7-adb9b877db9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" podUID="15316213-a694-41a6-aad7-adb9b877db9b" Nov 03 20:45:08.244360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.244343 5815 scope.go:122] "RemoveContainer" containerID="4f20fe1833ed81b3c74c8c7470b81875fbaea6fbb060afe3748903fee1113ebf" Nov 03 20:45:08.259422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.259409 5815 scope.go:122] "RemoveContainer" containerID="1ef04f2ff99eda173fdd1cb578529d13c19f142ebcd417695a0257b047ededb0" Nov 03 20:45:08.278153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.278134 5815 scope.go:122] "RemoveContainer" containerID="d5d2eb8ef0b1c4794425503ef2d0218bf5f49b9f709db2be14db4e226c2a376b" Nov 03 20:45:08.294669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:08.294647 5815 scope.go:122] "RemoveContainer" containerID="897defce3c145d49231515cc7d09aef4f47a6f85e19a68c09e37f0a3feb1c272" Nov 03 20:45:09.244185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:09.244095 5815 scope.go:122] "RemoveContainer" containerID="534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d" Nov 03 20:45:09.244648 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:09.244525 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6_odh-ai-gateway-infra(a9feecd9-d5d6-467c-8164-f8f9e6ba7b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" Nov 03 20:45:09.245963 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:09.245942 5815 scope.go:122] "RemoveContainer" containerID="d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd" Nov 03 20:45:09.246264 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:09.246241 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk_odh-ai-gateway-infra(d43955ae-3dcb-4b1b-9c57-ab375721ed29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" Nov 03 20:45:09.247983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:09.247963 5815 scope.go:122] "RemoveContainer" containerID="df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596" Nov 03 20:45:09.248277 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:09.248249 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl_odh-ai-gateway-infra(0264611c-c245-45a0-bef1-db5ae3b23fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" Nov 03 20:45:09.249726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:09.249710 5815 scope.go:122] "RemoveContainer" containerID="35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8" Nov 03 20:45:09.250001 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:09.249981 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf_odh-ai-gateway-infra(afad8ca6-5d95-4f58-9b90-5cf22546bb9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" Nov 03 20:45:09.251411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:09.251394 5815 scope.go:122] "RemoveContainer" containerID="3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409" Nov 03 20:45:09.251686 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:09.251664 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq_odh-ai-gateway-infra(1492ceb7-ca41-4a12-bc1b-d621d4a3118d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" Nov 03 20:45:09.253080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:09.253065 5815 scope.go:122] "RemoveContainer" containerID="4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df" Nov 03 20:45:09.253298 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:09.253281 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k_odh-ai-gateway-infra(81040066-6649-49bb-ac78-a1fccda7b2f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" Nov 03 20:45:13.853435 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:13.853383 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:45:17.028045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.028011 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt"] Nov 03 20:45:17.033871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.033834 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d520329e-qhjpt"] Nov 03 20:45:17.578120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.578091 5815 scope.go:122] "RemoveContainer" containerID="a163570ea19a6f6919772e0e0508ac9f7d85de54606c5ea5d57685b49da90e08" Nov 03 20:45:17.578357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.578295 5815 scope.go:122] "RemoveContainer" containerID="07fa560f39eae4cb692684ab28a79bb5efda399c87dea8f83cdf37aeb8c020d2" Nov 03 20:45:17.578500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.578479 5815 scope.go:122] "RemoveContainer" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" Nov 03 20:45:17.578856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.578833 5815 scope.go:122] "RemoveContainer" containerID="5379f109da6fbd232302761c84956a7183767361b0d94b5c00b96ecef4159517" Nov 03 20:45:17.578987 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.578967 5815 scope.go:122] "RemoveContainer" containerID="bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea" Nov 03 20:45:17.579802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.579086 5815 scope.go:122] "RemoveContainer" containerID="ecd2b80279c7919bd54b36d1aaf69ad6defaa8f39c2249bbf22b209c1d3fe730" Nov 03 20:45:17.579802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.579343 5815 scope.go:122] "RemoveContainer" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" Nov 03 20:45:17.579802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:17.579670 5815 scope.go:122] "RemoveContainer" containerID="f16c0213aeecc84197abe5de99515315d33da951790ac1f87ab135a987cc0e39" Nov 03 20:45:18.416531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.408575 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerStarted","Data":"d22b784e1b8877a56bc6a0786e386dc553429d84c6e3d9bfefc06cc05c1d427c"} Nov 03 20:45:18.416531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.412395 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerStarted","Data":"5a56c9170e683db53b40136730678fd7b18f92c8ce54a09ebb1bfac7ca83eb6c"} Nov 03 20:45:18.417052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.416541 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerStarted","Data":"3fe016b8e0dd15d7682c30d67502105f20d11d9b8ad6efc061c36b5b8dfb5e3d"} Nov 03 20:45:18.418798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.418774 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" event={"ID":"d54df193-ba22-440a-ad8c-28f8a9a93f32","Type":"ContainerStarted","Data":"8c328f6eed5da603560b538d5b2a25d0936490b8b6426589622606ca63db78a6"} Nov 03 20:45:18.421106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.420597 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerStarted","Data":"3bd0c8478ff6fc4ce66b10e44a83ba425843a19676c2085864345b4e5b500589"} Nov 03 20:45:18.422651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.422632 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerStarted","Data":"40b97ca5a410e36018cbf079ef0d1437aab784171a7a86e6541703e597916de8"} Nov 03 20:45:18.425173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.425152 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerStarted","Data":"1ab741593f8870be7d4b96526c323a90b4663f570dfb8f7085164b72fba8bce6"} Nov 03 20:45:18.428108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.428086 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" event={"ID":"24929273-c9e0-4a7f-9ad4-4d23cc831231","Type":"ContainerStarted","Data":"09283cf5424ea5a0a665caf96c1179b6c8bc312e6165f59ba4d5ecdb887e8439"} Nov 03 20:45:18.547360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.547091 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" podStartSLOduration=18.547074021 podStartE2EDuration="18.547074021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:18.543338645 +0000 UTC m=+2628.073684331" watchObservedRunningTime="2026-11-03 20:45:18.547074021 +0000 UTC m=+2628.077419692" Nov 03 20:45:18.578540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.578492 5815 scope.go:122] "RemoveContainer" containerID="3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409" Nov 03 20:45:18.580017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.579496 5815 scope.go:122] "RemoveContainer" containerID="35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8" Nov 03 20:45:18.602715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.601365 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" podStartSLOduration=18.601346705 podStartE2EDuration="18.601346705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:18.581500789 +0000 UTC m=+2628.111846458" watchObservedRunningTime="2026-11-03 20:45:18.601346705 +0000 UTC m=+2628.131692379" Nov 03 20:45:18.632870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:18.632833 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77" path="/var/lib/kubelet/pods/5d46ab80-d9fb-4311-9ae8-0eb9d74a2c77/volumes" Nov 03 20:45:19.379666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.377042 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft"] Nov 03 20:45:19.393631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.393588 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.411600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.411521 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft"] Nov 03 20:45:19.412979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.412952 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d4fe2\"/\"kube-root-ca.crt\"" Nov 03 20:45:19.414572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.414381 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d4fe2\"/\"default-dockercfg-ngxh9\"" Nov 03 20:45:19.414572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.414428 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d4fe2\"/\"openshift-service-ca.crt\"" Nov 03 20:45:19.414990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.414970 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d4fe2\"/\"test-model-3d4fe2-kserve-self-signed-certs\"" Nov 03 20:45:19.430396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.429483 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.430396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.429549 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.430396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.429606 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.430396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.429629 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrshs\" (UniqueName: \"kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.430396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.429715 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.430396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.429891 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.449838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.449811 5815 generic.go:356] "Generic (PLEG): container finished" podID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerID="3fe016b8e0dd15d7682c30d67502105f20d11d9b8ad6efc061c36b5b8dfb5e3d" exitCode=60 Nov 03 20:45:19.451349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.449887 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerDied","Data":"3fe016b8e0dd15d7682c30d67502105f20d11d9b8ad6efc061c36b5b8dfb5e3d"} Nov 03 20:45:19.451349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.450985 5815 scope.go:122] "RemoveContainer" containerID="5379f109da6fbd232302761c84956a7183767361b0d94b5c00b96ecef4159517" Nov 03 20:45:19.451830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.451801 5815 scope.go:122] "RemoveContainer" containerID="3fe016b8e0dd15d7682c30d67502105f20d11d9b8ad6efc061c36b5b8dfb5e3d" Nov 03 20:45:19.453580 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:19.452220 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf_odh-ai-gateway-infra(b3eb1e83-75bd-4701-82fc-98d2742904c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" Nov 03 20:45:19.454323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.454299 5815 generic.go:356] "Generic (PLEG): container finished" podID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerID="8c328f6eed5da603560b538d5b2a25d0936490b8b6426589622606ca63db78a6" exitCode=60 Nov 03 20:45:19.454452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.454428 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" event={"ID":"d54df193-ba22-440a-ad8c-28f8a9a93f32","Type":"ContainerDied","Data":"8c328f6eed5da603560b538d5b2a25d0936490b8b6426589622606ca63db78a6"} Nov 03 20:45:19.454811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.454675 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm"] Nov 03 20:45:19.454922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.454888 5815 scope.go:122] "RemoveContainer" containerID="8c328f6eed5da603560b538d5b2a25d0936490b8b6426589622606ca63db78a6" Nov 03 20:45:19.455282 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:19.455258 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l_odh-ai-gateway-infra(d54df193-ba22-440a-ad8c-28f8a9a93f32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" Nov 03 20:45:19.458260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.458236 5815 generic.go:356] "Generic (PLEG): container finished" podID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerID="3bd0c8478ff6fc4ce66b10e44a83ba425843a19676c2085864345b4e5b500589" exitCode=60 Nov 03 20:45:19.458369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.458262 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerDied","Data":"3bd0c8478ff6fc4ce66b10e44a83ba425843a19676c2085864345b4e5b500589"} Nov 03 20:45:19.458705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.458684 5815 scope.go:122] "RemoveContainer" containerID="3bd0c8478ff6fc4ce66b10e44a83ba425843a19676c2085864345b4e5b500589" Nov 03 20:45:19.459136 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:19.459095 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8_odh-ai-gateway-infra(d8f86469-cd81-4245-87c2-c487b3fc21f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" Nov 03 20:45:19.465042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.465016 5815 generic.go:356] "Generic (PLEG): container finished" podID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerID="40b97ca5a410e36018cbf079ef0d1437aab784171a7a86e6541703e597916de8" exitCode=60 Nov 03 20:45:19.465157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.465136 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerDied","Data":"40b97ca5a410e36018cbf079ef0d1437aab784171a7a86e6541703e597916de8"} Nov 03 20:45:19.465566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.465544 5815 scope.go:122] "RemoveContainer" containerID="40b97ca5a410e36018cbf079ef0d1437aab784171a7a86e6541703e597916de8" Nov 03 20:45:19.466094 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:19.466068 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq_odh-ai-gateway-infra(84ae2e98-dfd4-4a54-9546-c1cdf14c601a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" Nov 03 20:45:19.474104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.473165 5815 generic.go:356] "Generic (PLEG): container finished" podID="15316213-a694-41a6-aad7-adb9b877db9b" containerID="1ab741593f8870be7d4b96526c323a90b4663f570dfb8f7085164b72fba8bce6" exitCode=60 Nov 03 20:45:19.474389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.473186 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerDied","Data":"1ab741593f8870be7d4b96526c323a90b4663f570dfb8f7085164b72fba8bce6"} Nov 03 20:45:19.479338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.477996 5815 generic.go:356] "Generic (PLEG): container finished" podID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerID="09283cf5424ea5a0a665caf96c1179b6c8bc312e6165f59ba4d5ecdb887e8439" exitCode=60 Nov 03 20:45:19.479338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.478117 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" event={"ID":"24929273-c9e0-4a7f-9ad4-4d23cc831231","Type":"ContainerDied","Data":"09283cf5424ea5a0a665caf96c1179b6c8bc312e6165f59ba4d5ecdb887e8439"} Nov 03 20:45:19.479338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.478563 5815 scope.go:122] "RemoveContainer" containerID="09283cf5424ea5a0a665caf96c1179b6c8bc312e6165f59ba4d5ecdb887e8439" Nov 03 20:45:19.479338 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:19.479005 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t_odh-ai-gateway-infra(24929273-c9e0-4a7f-9ad4-4d23cc831231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" Nov 03 20:45:19.481193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.481176 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh"] Nov 03 20:45:19.484194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.483863 5815 generic.go:356] "Generic (PLEG): container finished" podID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerID="d22b784e1b8877a56bc6a0786e386dc553429d84c6e3d9bfefc06cc05c1d427c" exitCode=60 Nov 03 20:45:19.484194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.483967 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerDied","Data":"d22b784e1b8877a56bc6a0786e386dc553429d84c6e3d9bfefc06cc05c1d427c"} Nov 03 20:45:19.489384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.487899 5815 generic.go:356] "Generic (PLEG): container finished" podID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerID="5a56c9170e683db53b40136730678fd7b18f92c8ce54a09ebb1bfac7ca83eb6c" exitCode=60 Nov 03 20:45:19.489384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.488003 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerDied","Data":"5a56c9170e683db53b40136730678fd7b18f92c8ce54a09ebb1bfac7ca83eb6c"} Nov 03 20:45:19.490274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.490256 5815 scope.go:122] "RemoveContainer" containerID="1ab741593f8870be7d4b96526c323a90b4663f570dfb8f7085164b72fba8bce6" Nov 03 20:45:19.490969 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:19.490946 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx_odh-ai-gateway-infra(15316213-a694-41a6-aad7-adb9b877db9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" podUID="15316213-a694-41a6-aad7-adb9b877db9b" Nov 03 20:45:19.497838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.497648 5815 scope.go:122] "RemoveContainer" containerID="ecd2b80279c7919bd54b36d1aaf69ad6defaa8f39c2249bbf22b209c1d3fe730" Nov 03 20:45:19.531566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.531530 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.531706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.531582 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrshs\" (UniqueName: \"kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.531706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.531691 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.531969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.531952 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.535570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.532302 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.535570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.532336 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.543339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.537409 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.543339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.538136 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.544170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.544090 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.544170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.544122 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.546608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.546481 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:19.547918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.547840 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf"] Nov 03 20:45:19.682050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.654702 5815 scope.go:122] "RemoveContainer" containerID="24050d47e08c5b755d05072410578a870563b7b69e036be288588e2011a0340b" Nov 03 20:45:19.682050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.662925 5815 scope.go:122] "RemoveContainer" containerID="4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df" Nov 03 20:45:19.682050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.667892 5815 scope.go:122] "RemoveContainer" containerID="30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1" Nov 03 20:45:19.702012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.699767 5815 scope.go:122] "RemoveContainer" containerID="d6eb4c7181cb7b5ea3adab5fc4d40dcf306a2b7d461f9b882fe537d04e6d5fbb" Nov 03 20:45:19.702012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.700604 5815 scope.go:122] "RemoveContainer" containerID="d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd" Nov 03 20:45:19.703645 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.703562 5815 scope.go:122] "RemoveContainer" containerID="07fa560f39eae4cb692684ab28a79bb5efda399c87dea8f83cdf37aeb8c020d2" Nov 03 20:45:19.731548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.720283 5815 scope.go:122] "RemoveContainer" containerID="5f8ffeb1269029e9c646c3a2b7bb000b2e65a85e2628ba170a3ab651ed0044f4" Nov 03 20:45:19.951330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:19.951207 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrshs\" (UniqueName: \"kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs\") pod \"test-model-3d4fe2-kserve-7fc8f74dc9-8dcft\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:20.038585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.028394 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq"] Nov 03 20:45:20.040942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.040823 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:20.056429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.056397 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8"] Nov 03 20:45:20.079449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.079347 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l"] Nov 03 20:45:20.089635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.089575 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx"] Nov 03 20:45:20.117371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.117249 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t"] Nov 03 20:45:20.195700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.195396 5815 scope.go:122] "RemoveContainer" containerID="bbe08d9134fe9ba522d4660c4a7029b068bc79b6237bb4ff80304cc7e8e4e1ea" Nov 03 20:45:20.507989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.507957 5815 generic.go:356] "Generic (PLEG): container finished" podID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerID="0091d70f40ef540bb1c67d442091fc95348154ed453b21922a17831020a24e45" exitCode=60 Nov 03 20:45:20.508313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.508058 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" event={"ID":"afad8ca6-5d95-4f58-9b90-5cf22546bb9f","Type":"ContainerDied","Data":"0091d70f40ef540bb1c67d442091fc95348154ed453b21922a17831020a24e45"} Nov 03 20:45:20.508612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.508589 5815 scope.go:122] "RemoveContainer" containerID="0091d70f40ef540bb1c67d442091fc95348154ed453b21922a17831020a24e45" Nov 03 20:45:20.509074 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:20.509028 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf_odh-ai-gateway-infra(afad8ca6-5d95-4f58-9b90-5cf22546bb9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" Nov 03 20:45:20.511785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.511756 5815 generic.go:356] "Generic (PLEG): container finished" podID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerID="d20c778c221608558bf3415f5f2c23406a7f8b74753dee26c1cd5adedff49e61" exitCode=60 Nov 03 20:45:20.511933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.511912 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerDied","Data":"d20c778c221608558bf3415f5f2c23406a7f8b74753dee26c1cd5adedff49e61"} Nov 03 20:45:20.512342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.512323 5815 scope.go:122] "RemoveContainer" containerID="d20c778c221608558bf3415f5f2c23406a7f8b74753dee26c1cd5adedff49e61" Nov 03 20:45:20.512722 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:20.512701 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq_odh-ai-gateway-infra(1492ceb7-ca41-4a12-bc1b-d621d4a3118d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" Nov 03 20:45:20.583583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.582193 5815 scope.go:122] "RemoveContainer" containerID="df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596" Nov 03 20:45:20.758974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.756593 5815 scope.go:122] "RemoveContainer" containerID="f16c0213aeecc84197abe5de99515315d33da951790ac1f87ab135a987cc0e39" Nov 03 20:45:20.815579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:20.815547 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:45:21.376604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.357423 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:21.376604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.360359 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:21.414295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.414238 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca\") pod \"b508ca37-a709-402b-b160-d9f71c5e1aa3\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " Nov 03 20:45:21.414440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.414421 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp\" (UniqueName: \"kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp\") pod \"c5567729-6f60-4c64-b64d-9f032db9ebe6\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " Nov 03 20:45:21.418297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.418098 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca" pod "b508ca37-a709-402b-b160-d9f71c5e1aa3" (UID: "b508ca37-a709-402b-b160-d9f71c5e1aa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:21.419920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.418590 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r\" (UniqueName: \"kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r\") pod \"b508ca37-a709-402b-b160-d9f71c5e1aa3\" (UID: \"b508ca37-a709-402b-b160-d9f71c5e1aa3\") " Nov 03 20:45:21.419920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.418715 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca\") pod \"c5567729-6f60-4c64-b64d-9f032db9ebe6\" (UID: \"c5567729-6f60-4c64-b64d-9f032db9ebe6\") " Nov 03 20:45:21.420204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.420115 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b508ca37-a709-402b-b160-d9f71c5e1aa3-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.422528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.420435 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca" pod "c5567729-6f60-4c64-b64d-9f032db9ebe6" (UID: "c5567729-6f60-4c64-b64d-9f032db9ebe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:21.422528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.420944 5815 scope.go:122] "RemoveContainer" containerID="a163570ea19a6f6919772e0e0508ac9f7d85de54606c5ea5d57685b49da90e08" Nov 03 20:45:21.508043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.487086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r" pod "b508ca37-a709-402b-b160-d9f71c5e1aa3" (UID: "b508ca37-a709-402b-b160-d9f71c5e1aa3"). InnerVolumeSpecName "kube-api-access-t5h5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:21.523807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.523740 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp" pod "c5567729-6f60-4c64-b64d-9f032db9ebe6" (UID: "c5567729-6f60-4c64-b64d-9f032db9ebe6"). InnerVolumeSpecName "kube-api-access-jrwsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:21.528305 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.528282 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrwsp\" (UniqueName: \"kubernetes.io/projected/c5567729-6f60-4c64-b64d-9f032db9ebe6-kube-api-access-jrwsp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.528432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.528421 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5h5r\" (UniqueName: \"kubernetes.io/projected/b508ca37-a709-402b-b160-d9f71c5e1aa3-kube-api-access-t5h5r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.528540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.528526 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5567729-6f60-4c64-b64d-9f032db9ebe6-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.559696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.559441 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerStarted","Data":"80f8e9313bedb223355ccb82ea780489786ea2e9517670ef012241f446b26c01"} Nov 03 20:45:21.573593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.573533 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerStarted","Data":"3877ca9b8935331629849875e6e91dfa0ff07d1d1b8353f68c8627b24339f1da"} Nov 03 20:45:21.578769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.578748 5815 scope.go:122] "RemoveContainer" containerID="534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d" Nov 03 20:45:21.598049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.597062 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerStarted","Data":"969b70e0810d2b68d0a0e5b09b0604dca1dcd37148c239d52c90e43a601977ac"} Nov 03 20:45:21.626078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.625979 5815 scope.go:122] "RemoveContainer" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" Nov 03 20:45:21.626905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.626771 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:21.739972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.734975 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx\" (UniqueName: \"kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx\") pod \"d8f86469-cd81-4245-87c2-c487b3fc21f0\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " Nov 03 20:45:21.739972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.735261 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca\") pod \"d8f86469-cd81-4245-87c2-c487b3fc21f0\" (UID: \"d8f86469-cd81-4245-87c2-c487b3fc21f0\") " Nov 03 20:45:21.739972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.738420 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" event={"ID":"b508ca37-a709-402b-b160-d9f71c5e1aa3","Type":"ContainerDied","Data":"ef4c568d9e279629bdbf12004b17391839df95af0c09e1de1e96c1eda4dee4da"} Nov 03 20:45:21.739972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.738537 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm" Nov 03 20:45:21.739972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.739706 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca" pod "d8f86469-cd81-4245-87c2-c487b3fc21f0" (UID: "d8f86469-cd81-4245-87c2-c487b3fc21f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:21.753697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.753596 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq"] Nov 03 20:45:21.808730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.796719 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx" pod "d8f86469-cd81-4245-87c2-c487b3fc21f0" (UID: "d8f86469-cd81-4245-87c2-c487b3fc21f0"). InnerVolumeSpecName "kube-api-access-46lwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:21.808730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.802875 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" Nov 03 20:45:21.808730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.803854 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh" event={"ID":"c5567729-6f60-4c64-b64d-9f032db9ebe6","Type":"ContainerDied","Data":"cc3171e35ef4626e0d69fef4469b3cc76d1dae12a22ffdcbf1c19e11a38cb43a"} Nov 03 20:45:21.808730 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.807980 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:21.839707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.839172 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca\") pod \"b3eb1e83-75bd-4701-82fc-98d2742904c2\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " Nov 03 20:45:21.839707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.839247 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv\" (UniqueName: \"kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv\") pod \"b3eb1e83-75bd-4701-82fc-98d2742904c2\" (UID: \"b3eb1e83-75bd-4701-82fc-98d2742904c2\") " Nov 03 20:45:21.840816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.840791 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8f86469-cd81-4245-87c2-c487b3fc21f0-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.840918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.840821 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46lwx\" (UniqueName: \"kubernetes.io/projected/d8f86469-cd81-4245-87c2-c487b3fc21f0-kube-api-access-46lwx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.904643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.849691 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca" pod "b3eb1e83-75bd-4701-82fc-98d2742904c2" (UID: "b3eb1e83-75bd-4701-82fc-98d2742904c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:21.904643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.898949 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:21.910337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.905290 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf"] Nov 03 20:45:21.916400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.916081 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv" pod "b3eb1e83-75bd-4701-82fc-98d2742904c2" (UID: "b3eb1e83-75bd-4701-82fc-98d2742904c2"). InnerVolumeSpecName "kube-api-access-48hpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:21.935199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.934699 5815 scope.go:122] "RemoveContainer" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" Nov 03 20:45:21.940426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.940321 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:21.943480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.943459 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca\") pod \"15316213-a694-41a6-aad7-adb9b877db9b\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " Nov 03 20:45:21.944235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.943941 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca" pod "15316213-a694-41a6-aad7-adb9b877db9b" (UID: "15316213-a694-41a6-aad7-adb9b877db9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:21.944376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.944358 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4\" (UniqueName: \"kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4\") pod \"15316213-a694-41a6-aad7-adb9b877db9b\" (UID: \"15316213-a694-41a6-aad7-adb9b877db9b\") " Nov 03 20:45:21.946373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.946329 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15316213-a694-41a6-aad7-adb9b877db9b-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.966740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.946357 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb1e83-75bd-4701-82fc-98d2742904c2-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.966740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:21.954224 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hpv\" (UniqueName: \"kubernetes.io/projected/b3eb1e83-75bd-4701-82fc-98d2742904c2-kube-api-access-48hpv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.014787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.014744 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4" pod "15316213-a694-41a6-aad7-adb9b877db9b" (UID: "15316213-a694-41a6-aad7-adb9b877db9b"). InnerVolumeSpecName "kube-api-access-s6wr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:22.110425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.072940 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca\") pod \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " Nov 03 20:45:22.110425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.073289 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd\" (UniqueName: \"kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd\") pod \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\" (UID: \"84ae2e98-dfd4-4a54-9546-c1cdf14c601a\") " Nov 03 20:45:22.136530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.112410 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6wr4\" (UniqueName: \"kubernetes.io/projected/15316213-a694-41a6-aad7-adb9b877db9b-kube-api-access-s6wr4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.356803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.297053 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca" pod "84ae2e98-dfd4-4a54-9546-c1cdf14c601a" (UID: "84ae2e98-dfd4-4a54-9546-c1cdf14c601a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:22.356803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.331803 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.356803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.344118 5815 scope.go:122] "RemoveContainer" containerID="35821d3dc33aa9fc32b9e124eaf6eb6a72e71471c43eb9702ee910790d4e58e8" Nov 03 20:45:22.395850 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.395789 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:22.415373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.415302 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm"] Nov 03 20:45:22.435613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.435585 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895645-k9stm"] Nov 03 20:45:22.436475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.436406 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca\") pod \"24929273-c9e0-4a7f-9ad4-4d23cc831231\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " Nov 03 20:45:22.436956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.436935 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27\" (UniqueName: \"kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27\") pod \"24929273-c9e0-4a7f-9ad4-4d23cc831231\" (UID: \"24929273-c9e0-4a7f-9ad4-4d23cc831231\") " Nov 03 20:45:22.441109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.440912 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27" pod "24929273-c9e0-4a7f-9ad4-4d23cc831231" (UID: "24929273-c9e0-4a7f-9ad4-4d23cc831231"). InnerVolumeSpecName "kube-api-access-hvl27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:22.441442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.441353 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca" pod "24929273-c9e0-4a7f-9ad4-4d23cc831231" (UID: "24929273-c9e0-4a7f-9ad4-4d23cc831231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:22.461119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.461072 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh"] Nov 03 20:45:22.476901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.476813 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd" pod "84ae2e98-dfd4-4a54-9546-c1cdf14c601a" (UID: "84ae2e98-dfd4-4a54-9546-c1cdf14c601a"). InnerVolumeSpecName "kube-api-access-qh2sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:22.479529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.479457 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895645-v6cqh"] Nov 03 20:45:22.492480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.490603 5815 scope.go:122] "RemoveContainer" containerID="3562cd40f053c50103cfd29dc7a292ca47627a47facfef9904ca7694f3e6a409" Nov 03 20:45:22.505731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.505707 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:22.521133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.521022 5815 scope.go:122] "RemoveContainer" containerID="d22b784e1b8877a56bc6a0786e386dc553429d84c6e3d9bfefc06cc05c1d427c" Nov 03 20:45:22.540990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.540959 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24929273-c9e0-4a7f-9ad4-4d23cc831231-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.541179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.541165 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh2sd\" (UniqueName: \"kubernetes.io/projected/84ae2e98-dfd4-4a54-9546-c1cdf14c601a-kube-api-access-qh2sd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.541265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.541255 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvl27\" (UniqueName: \"kubernetes.io/projected/24929273-c9e0-4a7f-9ad4-4d23cc831231-kube-api-access-hvl27\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.546953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.546933 5815 scope.go:122] "RemoveContainer" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" Nov 03 20:45:22.547566 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:22.547479 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea\": container with ID starting with 61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea not found: ID does not exist" containerID="61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea" Nov 03 20:45:22.547722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.547581 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea"} err="failed to get container status \"61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea\": rpc error: code = NotFound desc = could not find container \"61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea\": container with ID starting with 61a8133ad9c640a4a571078322197b16d61135813ed6ce5f6a01efe73d02beea not found: ID does not exist" Nov 03 20:45:22.547722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.547608 5815 scope.go:122] "RemoveContainer" containerID="5a56c9170e683db53b40136730678fd7b18f92c8ce54a09ebb1bfac7ca83eb6c" Nov 03 20:45:22.571902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.571777 5815 scope.go:122] "RemoveContainer" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" Nov 03 20:45:22.573213 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:22.573108 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252\": container with ID starting with 409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252 not found: ID does not exist" containerID="409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252" Nov 03 20:45:22.573213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.573145 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252"} err="failed to get container status \"409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252\": rpc error: code = NotFound desc = could not find container \"409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252\": container with ID starting with 409acd3dd5f7baaa69f3b9d52b7f3af5ebe508932c1eeda75a1fe24e50d15252 not found: ID does not exist" Nov 03 20:45:22.621560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.616401 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" path="/var/lib/kubelet/pods/b508ca37-a709-402b-b160-d9f71c5e1aa3/volumes" Nov 03 20:45:22.631806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.631768 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" path="/var/lib/kubelet/pods/c5567729-6f60-4c64-b64d-9f032db9ebe6/volumes" Nov 03 20:45:22.644879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.644823 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca" pod "d54df193-ba22-440a-ad8c-28f8a9a93f32" (UID: "d54df193-ba22-440a-ad8c-28f8a9a93f32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:22.650395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.645171 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca\") pod \"d54df193-ba22-440a-ad8c-28f8a9a93f32\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " Nov 03 20:45:22.650395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.645555 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck\" (UniqueName: \"kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck\") pod \"d54df193-ba22-440a-ad8c-28f8a9a93f32\" (UID: \"d54df193-ba22-440a-ad8c-28f8a9a93f32\") " Nov 03 20:45:22.654202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.654166 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54df193-ba22-440a-ad8c-28f8a9a93f32-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.739534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.731246 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k"] Nov 03 20:45:22.872561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.846545 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5"] Nov 03 20:45:23.028098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:22.983134 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:23.130739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.114241 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck" pod "d54df193-ba22-440a-ad8c-28f8a9a93f32" (UID: "d54df193-ba22-440a-ad8c-28f8a9a93f32"). InnerVolumeSpecName "kube-api-access-2mwck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:23.173733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.172105 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca\") pod \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " Nov 03 20:45:23.173733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.172162 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql\" (UniqueName: \"kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql\") pod \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\" (UID: \"1492ceb7-ca41-4a12-bc1b-d621d4a3118d\") " Nov 03 20:45:23.173733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.173476 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwck\" (UniqueName: \"kubernetes.io/projected/d54df193-ba22-440a-ad8c-28f8a9a93f32-kube-api-access-2mwck\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:23.323959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.299978 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca" pod "1492ceb7-ca41-4a12-bc1b-d621d4a3118d" (UID: "1492ceb7-ca41-4a12-bc1b-d621d4a3118d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:23.323959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.302290 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:23.397810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.377735 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql" pod "1492ceb7-ca41-4a12-bc1b-d621d4a3118d" (UID: "1492ceb7-ca41-4a12-bc1b-d621d4a3118d"). InnerVolumeSpecName "kube-api-access-thkql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:23.472242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.415455 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thkql\" (UniqueName: \"kubernetes.io/projected/1492ceb7-ca41-4a12-bc1b-d621d4a3118d-kube-api-access-thkql\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:23.472242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.463958 5815 generic.go:356] "Generic (PLEG): container finished" podID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerID="31c79e4dcb918face2eddf03099d6c1b99edb0ff42360b3ea9e0c930f4a475f7" exitCode=60 Nov 03 20:45:23.472242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.464050 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" event={"ID":"9bd97fec-515e-4077-98ce-3257ff4316e1","Type":"ContainerDied","Data":"31c79e4dcb918face2eddf03099d6c1b99edb0ff42360b3ea9e0c930f4a475f7"} Nov 03 20:45:23.472242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.464105 5815 scope.go:122] "RemoveContainer" containerID="5f8ffeb1269029e9c646c3a2b7bb000b2e65a85e2628ba170a3ab651ed0044f4" Nov 03 20:45:23.472242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.464673 5815 scope.go:122] "RemoveContainer" containerID="31c79e4dcb918face2eddf03099d6c1b99edb0ff42360b3ea9e0c930f4a475f7" Nov 03 20:45:23.472242 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:23.465159 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z_odh-ai-gateway-infra(9bd97fec-515e-4077-98ce-3257ff4316e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" Nov 03 20:45:23.473244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.472915 5815 generic.go:356] "Generic (PLEG): container finished" podID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerID="969b70e0810d2b68d0a0e5b09b0604dca1dcd37148c239d52c90e43a601977ac" exitCode=60 Nov 03 20:45:23.473244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.473030 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerDied","Data":"969b70e0810d2b68d0a0e5b09b0604dca1dcd37148c239d52c90e43a601977ac"} Nov 03 20:45:23.493907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.488338 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" event={"ID":"15316213-a694-41a6-aad7-adb9b877db9b","Type":"ContainerDied","Data":"2671025f836bd87925a63ebfea9621689e591df3cab9e98a8103e66a94733db7"} Nov 03 20:45:23.493907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.488471 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx" Nov 03 20:45:23.600247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.600219 5815 scope.go:122] "RemoveContainer" containerID="969b70e0810d2b68d0a0e5b09b0604dca1dcd37148c239d52c90e43a601977ac" Nov 03 20:45:23.601045 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:23.601016 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8_odh-ai-gateway-infra(9666a784-df78-4aaa-9bbe-c6204efd4705)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" Nov 03 20:45:23.615395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.615357 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:23.780642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.780607 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8"] Nov 03 20:45:23.781561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.781532 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm\" (UniqueName: \"kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm\") pod \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " Nov 03 20:45:23.781794 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.781777 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca\") pod \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\" (UID: \"afad8ca6-5d95-4f58-9b90-5cf22546bb9f\") " Nov 03 20:45:23.783196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.783162 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" event={"ID":"24929273-c9e0-4a7f-9ad4-4d23cc831231","Type":"ContainerDied","Data":"1d21d1afebc377c8f7bbb87a2656cd9c75a26f6010b4591a1f4233739978a9d5"} Nov 03 20:45:23.783455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.783431 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t" Nov 03 20:45:23.785136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.785114 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca" pod "afad8ca6-5d95-4f58-9b90-5cf22546bb9f" (UID: "afad8ca6-5d95-4f58-9b90-5cf22546bb9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:23.845593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.829710 5815 scope.go:122] "RemoveContainer" containerID="30db7ccbba12069c93720e28335a58a160872122ab4b7a2531995dfb24ea4fd1" Nov 03 20:45:23.851249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.851051 5815 generic.go:356] "Generic (PLEG): container finished" podID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerID="1274a2927f4c0ade889e9d603cd12ab0823462b1975d690d9e062301c897e232" exitCode=60 Nov 03 20:45:23.851341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.851247 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" event={"ID":"0264611c-c245-45a0-bef1-db5ae3b23fdf","Type":"ContainerDied","Data":"1274a2927f4c0ade889e9d603cd12ab0823462b1975d690d9e062301c897e232"} Nov 03 20:45:23.861119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.856069 5815 scope.go:122] "RemoveContainer" containerID="1274a2927f4c0ade889e9d603cd12ab0823462b1975d690d9e062301c897e232" Nov 03 20:45:23.914533 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:23.902833 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl_odh-ai-gateway-infra(0264611c-c245-45a0-bef1-db5ae3b23fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" Nov 03 20:45:23.914533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.894903 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf" Nov 03 20:45:23.914533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.857159 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm" pod "afad8ca6-5d95-4f58-9b90-5cf22546bb9f" (UID: "afad8ca6-5d95-4f58-9b90-5cf22546bb9f"). InnerVolumeSpecName "kube-api-access-82pfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:23.979336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.961402 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pfm\" (UniqueName: \"kubernetes.io/projected/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-kube-api-access-82pfm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:23.979336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.961429 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afad8ca6-5d95-4f58-9b90-5cf22546bb9f-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:23.979336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.962669 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" event={"ID":"1492ceb7-ca41-4a12-bc1b-d621d4a3118d","Type":"ContainerDied","Data":"93a54e1c4781680138a6b0df732f05abd7b5870b4d1073b0f03d9cbbaaaf5a97"} Nov 03 20:45:23.979336 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:23.962774 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq" Nov 03 20:45:24.028627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.028345 5815 scope.go:122] "RemoveContainer" containerID="1ab741593f8870be7d4b96526c323a90b4663f570dfb8f7085164b72fba8bce6" Nov 03 20:45:24.034424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.033945 5815 generic.go:356] "Generic (PLEG): container finished" podID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerID="80f8e9313bedb223355ccb82ea780489786ea2e9517670ef012241f446b26c01" exitCode=60 Nov 03 20:45:24.034424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.034029 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerDied","Data":"80f8e9313bedb223355ccb82ea780489786ea2e9517670ef012241f446b26c01"} Nov 03 20:45:24.038357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.038200 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft"] Nov 03 20:45:24.039474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.039444 5815 generic.go:356] "Generic (PLEG): container finished" podID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerID="8008c37a4770b043f152a53f9afca6b4b1fb034695b293fdabcfdde85f8f4607" exitCode=60 Nov 03 20:45:24.039613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.039589 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" event={"ID":"ea08089a-857b-4432-8549-50ca4bf1dbbc","Type":"ContainerDied","Data":"8008c37a4770b043f152a53f9afca6b4b1fb034695b293fdabcfdde85f8f4607"} Nov 03 20:45:24.042896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.042876 5815 scope.go:122] "RemoveContainer" containerID="8008c37a4770b043f152a53f9afca6b4b1fb034695b293fdabcfdde85f8f4607" Nov 03 20:45:24.043556 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:24.043494 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g_odh-ai-gateway-infra(ea08089a-857b-4432-8549-50ca4bf1dbbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" Nov 03 20:45:24.045798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.045777 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" event={"ID":"b3eb1e83-75bd-4701-82fc-98d2742904c2","Type":"ContainerDied","Data":"66b66a28ae73b5e1544c555a8da6f92e8d3545d8e46c4f7db790a73756b4837b"} Nov 03 20:45:24.046025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.046009 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf" Nov 03 20:45:24.049597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.049568 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" Nov 03 20:45:24.062857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.062780 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l" event={"ID":"d54df193-ba22-440a-ad8c-28f8a9a93f32","Type":"ContainerDied","Data":"1c692e24145b9173d0cd33dd38aa8eea163565a4aef048b9ba13bc473dea88bc"} Nov 03 20:45:24.121931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.121907 5815 scope.go:122] "RemoveContainer" containerID="09283cf5424ea5a0a665caf96c1179b6c8bc312e6165f59ba4d5ecdb887e8439" Nov 03 20:45:24.122941 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.122924 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx"] Nov 03 20:45:24.123139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.123117 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerStarted","Data":"293613d9cc10213d95d2f25301745f2d1774af9b37413c5c9b48b7f7118d8e5c"} Nov 03 20:45:24.123953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.123935 5815 scope.go:122] "RemoveContainer" containerID="293613d9cc10213d95d2f25301745f2d1774af9b37413c5c9b48b7f7118d8e5c" Nov 03 20:45:24.124883 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:24.124856 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6_odh-ai-gateway-infra(a9feecd9-d5d6-467c-8164-f8f9e6ba7b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" Nov 03 20:45:24.127203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.127179 5815 generic.go:356] "Generic (PLEG): container finished" podID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerID="3877ca9b8935331629849875e6e91dfa0ff07d1d1b8353f68c8627b24339f1da" exitCode=60 Nov 03 20:45:24.127413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.127396 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerDied","Data":"3877ca9b8935331629849875e6e91dfa0ff07d1d1b8353f68c8627b24339f1da"} Nov 03 20:45:24.135978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.135135 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" Nov 03 20:45:24.144533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.139783 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8" event={"ID":"d8f86469-cd81-4245-87c2-c487b3fc21f0","Type":"ContainerDied","Data":"40b8ac70027cef144f01e0dae5593b1006c91c249e94f0c3742ffb28c0f63f27"} Nov 03 20:45:24.144533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.139827 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895645-h4dxx"] Nov 03 20:45:24.154339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.154192 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t"] Nov 03 20:45:24.189947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.186642 5815 scope.go:122] "RemoveContainer" containerID="df57d69329601b79355747f6bd10118050531f06e7c36a013e0ed9d94dd98596" Nov 03 20:45:24.204172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.204138 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895645-ljp4t"] Nov 03 20:45:24.212284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.212249 5815 generic.go:356] "Generic (PLEG): container finished" podID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerID="a02b22491efa394cd7b27996e16ff1d825e20fd64201cdf29298e69f74bfc039" exitCode=60 Nov 03 20:45:24.212467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.212391 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" event={"ID":"d43955ae-3dcb-4b1b-9c57-ab375721ed29","Type":"ContainerDied","Data":"a02b22491efa394cd7b27996e16ff1d825e20fd64201cdf29298e69f74bfc039"} Nov 03 20:45:24.214445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.213793 5815 scope.go:122] "RemoveContainer" containerID="a02b22491efa394cd7b27996e16ff1d825e20fd64201cdf29298e69f74bfc039" Nov 03 20:45:24.215363 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:24.214864 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk_odh-ai-gateway-infra(d43955ae-3dcb-4b1b-9c57-ab375721ed29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" Nov 03 20:45:24.218352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.218325 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" event={"ID":"84ae2e98-dfd4-4a54-9546-c1cdf14c601a","Type":"ContainerDied","Data":"815f4f2a0a3faddeaff70a7cc933ae41f4dd6ed41995f1906bcbf8db2113c8a0"} Nov 03 20:45:24.218496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.218442 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq" Nov 03 20:45:24.245566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.245486 5815 scope.go:122] "RemoveContainer" containerID="d20c778c221608558bf3415f5f2c23406a7f8b74753dee26c1cd5adedff49e61" Nov 03 20:45:24.308866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.308744 5815 scope.go:122] "RemoveContainer" containerID="4aa3a98551aa8ad96e51c01091e65aac7af1adc7ce40a018835d843e0bc9e1df" Nov 03 20:45:24.315271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.315249 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf"] Nov 03 20:45:24.326646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.326620 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895645-7brjf"] Nov 03 20:45:24.344494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.344470 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq"] Nov 03 20:45:24.350750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.350730 5815 scope.go:122] "RemoveContainer" containerID="24050d47e08c5b755d05072410578a870563b7b69e036be288588e2011a0340b" Nov 03 20:45:24.353442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.353412 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895645-9vkdq"] Nov 03 20:45:24.366236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.366165 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8"] Nov 03 20:45:24.377786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.377722 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-03c633-29895645-lgwq8"] Nov 03 20:45:24.387388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.387319 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l"] Nov 03 20:45:24.408983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.408756 5815 scope.go:122] "RemoveContainer" containerID="3fe016b8e0dd15d7682c30d67502105f20d11d9b8ad6efc061c36b5b8dfb5e3d" Nov 03 20:45:24.409468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.409404 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895645-gpq7l"] Nov 03 20:45:24.435608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.435585 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq"] Nov 03 20:45:24.510634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.510596 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d8d0fa-29895645-fn7rq"] Nov 03 20:45:24.532825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.532794 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf"] Nov 03 20:45:24.539154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.539129 5815 scope.go:122] "RemoveContainer" containerID="8c328f6eed5da603560b538d5b2a25d0936490b8b6426589622606ca63db78a6" Nov 03 20:45:24.572538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.566410 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:24.652983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.647035 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88\" (UniqueName: \"kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88\") pod \"81040066-6649-49bb-ac78-a1fccda7b2f5\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " Nov 03 20:45:24.652983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.647127 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca\") pod \"81040066-6649-49bb-ac78-a1fccda7b2f5\" (UID: \"81040066-6649-49bb-ac78-a1fccda7b2f5\") " Nov 03 20:45:24.659011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.658910 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca" pod "81040066-6649-49bb-ac78-a1fccda7b2f5" (UID: "81040066-6649-49bb-ac78-a1fccda7b2f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:24.708367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.708325 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81040066-6649-49bb-ac78-a1fccda7b2f5-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:24.734275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.734245 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" path="/var/lib/kubelet/pods/1492ceb7-ca41-4a12-bc1b-d621d4a3118d/volumes" Nov 03 20:45:24.735377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.735348 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15316213-a694-41a6-aad7-adb9b877db9b" path="/var/lib/kubelet/pods/15316213-a694-41a6-aad7-adb9b877db9b/volumes" Nov 03 20:45:24.736883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.736101 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" path="/var/lib/kubelet/pods/24929273-c9e0-4a7f-9ad4-4d23cc831231/volumes" Nov 03 20:45:24.737287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.737263 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" path="/var/lib/kubelet/pods/84ae2e98-dfd4-4a54-9546-c1cdf14c601a/volumes" Nov 03 20:45:24.738098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.738074 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" path="/var/lib/kubelet/pods/b3eb1e83-75bd-4701-82fc-98d2742904c2/volumes" Nov 03 20:45:24.739297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.739273 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" path="/var/lib/kubelet/pods/d54df193-ba22-440a-ad8c-28f8a9a93f32/volumes" Nov 03 20:45:24.739746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.739689 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88" pod "81040066-6649-49bb-ac78-a1fccda7b2f5" (UID: "81040066-6649-49bb-ac78-a1fccda7b2f5"). InnerVolumeSpecName "kube-api-access-f9h88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:24.742557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.740582 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" path="/var/lib/kubelet/pods/d8f86469-cd81-4245-87c2-c487b3fc21f0/volumes" Nov 03 20:45:24.742557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.741350 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895645-4rbzf"] Nov 03 20:45:24.742557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.741389 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z"] Nov 03 20:45:24.793747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.793409 5815 scope.go:122] "RemoveContainer" containerID="d6eb4c7181cb7b5ea3adab5fc4d40dcf306a2b7d461f9b882fe537d04e6d5fbb" Nov 03 20:45:24.810326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.810288 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9h88\" (UniqueName: \"kubernetes.io/projected/81040066-6649-49bb-ac78-a1fccda7b2f5-kube-api-access-f9h88\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:24.813669 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.813643 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:24.830426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.830270 5815 scope.go:122] "RemoveContainer" containerID="3bd0c8478ff6fc4ce66b10e44a83ba425843a19676c2085864345b4e5b500589" Nov 03 20:45:24.849309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.849286 5815 scope.go:122] "RemoveContainer" containerID="d429870ded36aebbfde7434a2a2ecbb552f9ad724d4f9d70869dfa31823476bd" Nov 03 20:45:24.901623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.901598 5815 scope.go:122] "RemoveContainer" containerID="40b97ca5a410e36018cbf079ef0d1437aab784171a7a86e6541703e597916de8" Nov 03 20:45:24.914552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.914498 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca\") pod \"f035d917-9f8f-417e-9951-1bda4a7f6779\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " Nov 03 20:45:24.914706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.914578 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn\" (UniqueName: \"kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn\") pod \"f035d917-9f8f-417e-9951-1bda4a7f6779\" (UID: \"f035d917-9f8f-417e-9951-1bda4a7f6779\") " Nov 03 20:45:24.915258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.915238 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca" pod "f035d917-9f8f-417e-9951-1bda4a7f6779" (UID: "f035d917-9f8f-417e-9951-1bda4a7f6779"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:24.915674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.915653 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f035d917-9f8f-417e-9951-1bda4a7f6779-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:24.916951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.916928 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn" pod "f035d917-9f8f-417e-9951-1bda4a7f6779" (UID: "f035d917-9f8f-417e-9951-1bda4a7f6779"). InnerVolumeSpecName "kube-api-access-626hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:24.930984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:24.930961 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl"] Nov 03 20:45:25.017369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.017332 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626hn\" (UniqueName: \"kubernetes.io/projected/f035d917-9f8f-417e-9951-1bda4a7f6779-kube-api-access-626hn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:25.113824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.113789 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g"] Nov 03 20:45:25.232257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.232148 5815 generic.go:356] "Generic (PLEG): container finished" podID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerID="293613d9cc10213d95d2f25301745f2d1774af9b37413c5c9b48b7f7118d8e5c" exitCode=60 Nov 03 20:45:25.232257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.232226 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerDied","Data":"293613d9cc10213d95d2f25301745f2d1774af9b37413c5c9b48b7f7118d8e5c"} Nov 03 20:45:25.232500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.232296 5815 scope.go:122] "RemoveContainer" containerID="534bf6b1815e95f4a2b3fcca564aafee60679320c0c0d8d00dfe1ec3f87d785d" Nov 03 20:45:25.232762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.232743 5815 scope.go:122] "RemoveContainer" containerID="293613d9cc10213d95d2f25301745f2d1774af9b37413c5c9b48b7f7118d8e5c" Nov 03 20:45:25.233139 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:25.233118 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6_odh-ai-gateway-infra(a9feecd9-d5d6-467c-8164-f8f9e6ba7b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" Nov 03 20:45:25.238803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.238777 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" event={"ID":"f035d917-9f8f-417e-9951-1bda4a7f6779","Type":"ContainerDied","Data":"fe7307ae0e1026e3ca9cc70f4f319c0ebf07edeee69a33e257e2cc316d2ad5ee"} Nov 03 20:45:25.238940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.238860 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5" Nov 03 20:45:25.247717 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.247692 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6"] Nov 03 20:45:25.257905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.257853 5815 scope.go:122] "RemoveContainer" containerID="3877ca9b8935331629849875e6e91dfa0ff07d1d1b8353f68c8627b24339f1da" Nov 03 20:45:25.305255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.304296 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" event={"ID":"81040066-6649-49bb-ac78-a1fccda7b2f5","Type":"ContainerDied","Data":"427098de8140ed96d7c703c98414aa9783f5fa27bd5373c5e6227aee88fd514c"} Nov 03 20:45:25.305255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.304415 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k" Nov 03 20:45:25.328872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.328845 5815 scope.go:122] "RemoveContainer" containerID="0091d70f40ef540bb1c67d442091fc95348154ed453b21922a17831020a24e45" Nov 03 20:45:25.335195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.334424 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" event={"ID":"0c5c598f-2394-40f5-95e1-42c38a587203","Type":"ContainerStarted","Data":"8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6"} Nov 03 20:45:25.335195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.334463 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" event={"ID":"0c5c598f-2394-40f5-95e1-42c38a587203","Type":"ContainerStarted","Data":"9378fbff0840b8615a49f5c98a891004b563d8867369de8b49b9642904cd6920"} Nov 03 20:45:25.359785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.359755 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5"] Nov 03 20:45:25.379786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.377388 5815 scope.go:122] "RemoveContainer" containerID="80f8e9313bedb223355ccb82ea780489786ea2e9517670ef012241f446b26c01" Nov 03 20:45:25.398288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.398230 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895645-cvcq5"] Nov 03 20:45:25.846616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.846574 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk"] Nov 03 20:45:25.923897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.923269 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k"] Nov 03 20:45:25.935664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.935547 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895645-4xg8k"] Nov 03 20:45:25.958418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:25.958338 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podStartSLOduration=6.95831966 podStartE2EDuration="6.95831966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:25.512765532 +0000 UTC m=+2635.043111206" watchObservedRunningTime="2026-11-03 20:45:25.95831966 +0000 UTC m=+2635.488665334" Nov 03 20:45:26.043676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.043642 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:26.096113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.095731 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:26.315982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.246724 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc\" (UniqueName: \"kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc\") pod \"9666a784-df78-4aaa-9bbe-c6204efd4705\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " Nov 03 20:45:26.315982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.246891 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca\") pod \"9666a784-df78-4aaa-9bbe-c6204efd4705\" (UID: \"9666a784-df78-4aaa-9bbe-c6204efd4705\") " Nov 03 20:45:26.344322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.340402 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca" pod "9666a784-df78-4aaa-9bbe-c6204efd4705" (UID: "9666a784-df78-4aaa-9bbe-c6204efd4705"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:26.353263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.353207 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9666a784-df78-4aaa-9bbe-c6204efd4705-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.362968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.362896 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc" pod "9666a784-df78-4aaa-9bbe-c6204efd4705" (UID: "9666a784-df78-4aaa-9bbe-c6204efd4705"). InnerVolumeSpecName "kube-api-access-5wjxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:26.375744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.375364 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:26.454885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.454852 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw\" (UniqueName: \"kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw\") pod \"9bd97fec-515e-4077-98ce-3257ff4316e1\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " Nov 03 20:45:26.455091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.455069 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca\") pod \"9bd97fec-515e-4077-98ce-3257ff4316e1\" (UID: \"9bd97fec-515e-4077-98ce-3257ff4316e1\") " Nov 03 20:45:26.456040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.456010 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjxc\" (UniqueName: \"kubernetes.io/projected/9666a784-df78-4aaa-9bbe-c6204efd4705-kube-api-access-5wjxc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.456973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.456940 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca" pod "9bd97fec-515e-4077-98ce-3257ff4316e1" (UID: "9bd97fec-515e-4077-98ce-3257ff4316e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:26.457149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.457119 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" event={"ID":"9bd97fec-515e-4077-98ce-3257ff4316e1","Type":"ContainerDied","Data":"392d550495213b3f1faa2779187969234c6d66cd181818d20b8ca761d678087d"} Nov 03 20:45:26.457222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.457190 5815 scope.go:122] "RemoveContainer" containerID="31c79e4dcb918face2eddf03099d6c1b99edb0ff42360b3ea9e0c930f4a475f7" Nov 03 20:45:26.457295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.457279 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z" Nov 03 20:45:26.462281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.462240 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" event={"ID":"9666a784-df78-4aaa-9bbe-c6204efd4705","Type":"ContainerDied","Data":"cbb4964e23e9f4aeaf5715f314ae25af3cc5ed7bc45c7c9bff341a2c38870fe4"} Nov 03 20:45:26.463720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.463704 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8" Nov 03 20:45:26.531568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.531034 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw" pod "9bd97fec-515e-4077-98ce-3257ff4316e1" (UID: "9bd97fec-515e-4077-98ce-3257ff4316e1"). InnerVolumeSpecName "kube-api-access-bq6jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:26.538302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.537924 5815 scope.go:122] "RemoveContainer" containerID="969b70e0810d2b68d0a0e5b09b0604dca1dcd37148c239d52c90e43a601977ac" Nov 03 20:45:26.546138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.546091 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:26.554013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.553989 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:26.557603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.557581 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq\" (UniqueName: \"kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq\") pod \"0264611c-c245-45a0-bef1-db5ae3b23fdf\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " Nov 03 20:45:26.557867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.557849 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca\") pod \"0264611c-c245-45a0-bef1-db5ae3b23fdf\" (UID: \"0264611c-c245-45a0-bef1-db5ae3b23fdf\") " Nov 03 20:45:26.558205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.558182 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8"] Nov 03 20:45:26.558472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.558454 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd97fec-515e-4077-98ce-3257ff4316e1-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.558583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.558480 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6jw\" (UniqueName: \"kubernetes.io/projected/9bd97fec-515e-4077-98ce-3257ff4316e1-kube-api-access-bq6jw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.558856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.558824 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca" pod "0264611c-c245-45a0-bef1-db5ae3b23fdf" (UID: "0264611c-c245-45a0-bef1-db5ae3b23fdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:26.562544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.562483 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq" pod "0264611c-c245-45a0-bef1-db5ae3b23fdf" (UID: "0264611c-c245-45a0-bef1-db5ae3b23fdf"). InnerVolumeSpecName "kube-api-access-zm5sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:26.595572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.592262 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895645-mczn8"] Nov 03 20:45:26.632089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.632057 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" path="/var/lib/kubelet/pods/81040066-6649-49bb-ac78-a1fccda7b2f5/volumes" Nov 03 20:45:26.632661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.632638 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" path="/var/lib/kubelet/pods/9666a784-df78-4aaa-9bbe-c6204efd4705/volumes" Nov 03 20:45:26.633145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.633129 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" path="/var/lib/kubelet/pods/afad8ca6-5d95-4f58-9b90-5cf22546bb9f/volumes" Nov 03 20:45:26.634122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.634089 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" path="/var/lib/kubelet/pods/f035d917-9f8f-417e-9951-1bda4a7f6779/volumes" Nov 03 20:45:26.645273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.645238 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg"] Nov 03 20:45:26.646230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646198 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.646230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646220 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646236 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646244 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646254 5815 cpu_manager.go:414] "containerMap: removing container" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646260 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646278 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646285 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646308 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646315 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646327 5815 cpu_manager.go:414] "containerMap: removing container" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646334 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646345 5815 cpu_manager.go:414] "containerMap: removing container" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646351 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646360 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646366 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646377 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646384 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646395 5815 cpu_manager.go:414] "containerMap: removing container" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646403 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646413 5815 cpu_manager.go:414] "containerMap: removing container" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646421 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646435 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:26.646439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646442 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646468 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646475 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646486 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646493 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646518 5815 cpu_manager.go:414] "containerMap: removing container" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646525 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646538 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646544 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646556 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646563 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646573 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646581 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646589 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646596 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646611 5815 cpu_manager.go:414] "containerMap: removing container" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646617 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646631 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646638 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646651 5815 cpu_manager.go:414] "containerMap: removing container" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646658 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646673 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646679 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646689 5815 cpu_manager.go:414] "containerMap: removing container" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646695 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646704 5815 cpu_manager.go:414] "containerMap: removing container" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646711 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646726 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646732 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646745 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646752 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646764 5815 cpu_manager.go:414] "containerMap: removing container" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646772 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646797 5815 cpu_manager.go:414] "containerMap: removing container" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646804 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646816 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646822 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646834 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646841 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646856 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646863 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646874 5815 cpu_manager.go:414] "containerMap: removing container" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646881 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646892 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646897 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646909 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646916 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646929 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646936 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646955 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646962 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646972 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.646979 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647214 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647231 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647241 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647251 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647259 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647271 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647282 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647292 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647302 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647312 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647322 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647329 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647337 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647345 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647357 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647366 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647374 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647383 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647393 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647401 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647411 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="24929273-c9e0-4a7f-9ad4-4d23cc831231" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647422 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647433 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647441 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647452 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647461 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647470 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647480 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f86469-cd81-4245-87c2-c487b3fc21f0" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647490 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647498 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647521 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647532 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647540 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647548 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647561 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647570 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5567729-6f60-4c64-b64d-9f032db9ebe6" containerName="cleanup" Nov 03 20:45:26.648116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.647579 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:26.659171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.658854 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg"] Nov 03 20:45:26.659171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.658879 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.659700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.659675 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:26.659905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.659883 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2\") pod \"ea08089a-857b-4432-8549-50ca4bf1dbbc\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " Nov 03 20:45:26.660140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.660123 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca\") pod \"ea08089a-857b-4432-8549-50ca4bf1dbbc\" (UID: \"ea08089a-857b-4432-8549-50ca4bf1dbbc\") " Nov 03 20:45:26.660655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.660626 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca" pod "ea08089a-857b-4432-8549-50ca4bf1dbbc" (UID: "ea08089a-857b-4432-8549-50ca4bf1dbbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:26.661452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.661422 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0264611c-c245-45a0-bef1-db5ae3b23fdf-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.661602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.661588 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea08089a-857b-4432-8549-50ca4bf1dbbc-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.661691 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.661681 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5sq\" (UniqueName: \"kubernetes.io/projected/0264611c-c245-45a0-bef1-db5ae3b23fdf-kube-api-access-zm5sq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.685529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.673638 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:45:26.707647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.707603 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2" pod "ea08089a-857b-4432-8549-50ca4bf1dbbc" (UID: "ea08089a-857b-4432-8549-50ca4bf1dbbc"). InnerVolumeSpecName "kube-api-access-dc9s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:26.762295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.762217 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd\" (UniqueName: \"kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd\") pod \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " Nov 03 20:45:26.763086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.762310 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca\") pod \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\" (UID: \"d43955ae-3dcb-4b1b-9c57-ab375721ed29\") " Nov 03 20:45:26.763086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.762897 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca" pod "d43955ae-3dcb-4b1b-9c57-ab375721ed29" (UID: "d43955ae-3dcb-4b1b-9c57-ab375721ed29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:26.777535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.768389 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbvc\" (UniqueName: \"kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.777535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.768556 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.777535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.769072 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.777535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.769547 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc9s2\" (UniqueName: \"kubernetes.io/projected/ea08089a-857b-4432-8549-50ca4bf1dbbc-kube-api-access-dc9s2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.777535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.769571 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43955ae-3dcb-4b1b-9c57-ab375721ed29-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.807877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.807819 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd" pod "d43955ae-3dcb-4b1b-9c57-ab375721ed29" (UID: "d43955ae-3dcb-4b1b-9c57-ab375721ed29"). InnerVolumeSpecName "kube-api-access-rs9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:26.822392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.822326 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z"] Nov 03 20:45:26.828204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.828172 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895645-b5p2z"] Nov 03 20:45:26.875159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.875073 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbvc\" (UniqueName: \"kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.881613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.876173 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.882775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.882749 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.887264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.887245 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9xd\" (UniqueName: \"kubernetes.io/projected/d43955ae-3dcb-4b1b-9c57-ab375721ed29-kube-api-access-rs9xd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.887541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.887496 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.891228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.891180 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbvc\" (UniqueName: \"kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.904836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.904786 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d046963a-4jxtg\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:26.935088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.935069 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:27.003534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.988319 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts\" (UniqueName: \"kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts\") pod \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " Nov 03 20:45:27.003534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.988543 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca\") pod \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\" (UID: \"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03\") " Nov 03 20:45:27.003534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:26.989937 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca" pod "a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" (UID: "a9feecd9-d5d6-467c-8164-f8f9e6ba7b03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:27.004741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:27.004702 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:27.006084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:27.006058 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts" pod "a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" (UID: "a9feecd9-d5d6-467c-8164-f8f9e6ba7b03"). InnerVolumeSpecName "kube-api-access-9xpts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:27.040709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:27.040681 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:27.116540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:27.109498 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xpts\" (UniqueName: \"kubernetes.io/projected/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03-kube-api-access-9xpts\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:28.314855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.291930 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv"] Nov 03 20:45:28.336173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.336131 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" event={"ID":"0264611c-c245-45a0-bef1-db5ae3b23fdf","Type":"ContainerDied","Data":"06478a71e9bdf7c74295636b6fdbd92c591f5fe2d927710cfe9d0b23897187f4"} Nov 03 20:45:28.336448 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.336426 5815 scope.go:122] "RemoveContainer" containerID="1274a2927f4c0ade889e9d603cd12ab0823462b1975d690d9e062301c897e232" Nov 03 20:45:28.345020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.344992 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl" Nov 03 20:45:28.422442 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.422408 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" Nov 03 20:45:28.472534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.422212 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g" event={"ID":"ea08089a-857b-4432-8549-50ca4bf1dbbc","Type":"ContainerDied","Data":"1b12b62895018606cd4d1af2b836c798b9c94404608ed7b209f752da61167b68"} Nov 03 20:45:28.472828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.451691 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:28.529189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.478345 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:28.529189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.487932 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f0d73d-gbptv"] Nov 03 20:45:28.625593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.619559 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:28.751565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.734355 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" Nov 03 20:45:28.778586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.771011 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:28.959309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:28.959207 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:29.219752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.219703 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" Nov 03 20:45:29.224461 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.224437 5815 scope.go:122] "RemoveContainer" containerID="8008c37a4770b043f152a53f9afca6b4b1fb034695b293fdabcfdde85f8f4607" Nov 03 20:45:29.246978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.246943 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c903d0c-1aba-42bc-8c8e-b0d44cca353a" path="/var/lib/kubelet/pods/2c903d0c-1aba-42bc-8c8e-b0d44cca353a/volumes" Nov 03 20:45:29.248889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.248864 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" path="/var/lib/kubelet/pods/9bd97fec-515e-4077-98ce-3257ff4316e1/volumes" Nov 03 20:45:29.252286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.252266 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg"] Nov 03 20:45:29.254588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.252332 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw"] Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256444 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256463 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256477 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256485 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256537 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256547 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f035d917-9f8f-417e-9951-1bda4a7f6779" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256565 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256572 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54df193-ba22-440a-ad8c-28f8a9a93f32" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256605 5815 cpu_manager.go:414] "containerMap: removing container" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256614 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256625 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:29.256633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256632 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256645 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256653 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256664 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256671 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256704 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256711 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256728 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256734 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256750 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256757 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256768 5815 cpu_manager.go:414] "containerMap: removing container" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256774 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256786 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256794 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256805 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256812 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256827 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.256834 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257043 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257060 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3eb1e83-75bd-4701-82fc-98d2742904c2" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257074 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="81040066-6649-49bb-ac78-a1fccda7b2f5" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257085 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd97fec-515e-4077-98ce-3257ff4316e1" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257093 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257103 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257115 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1492ceb7-ca41-4a12-bc1b-d621d4a3118d" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257123 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257132 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b508ca37-a709-402b-b160-d9f71c5e1aa3" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257145 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="afad8ca6-5d95-4f58-9b90-5cf22546bb9f" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257158 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257166 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9666a784-df78-4aaa-9bbe-c6204efd4705" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257175 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="15316213-a694-41a6-aad7-adb9b877db9b" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257184 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257196 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.257348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.257208 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ae2e98-dfd4-4a54-9546-c1cdf14c601a" containerName="cleanup" Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.281372 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw"] Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.281419 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6" event={"ID":"a9feecd9-d5d6-467c-8164-f8f9e6ba7b03","Type":"ContainerDied","Data":"48314be9d14fd1065ad43b5dd3b0d7cf01667ba2397fbd5874da43743490298e"} Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.281455 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64"] Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.282550 5815 scope.go:122] "RemoveContainer" containerID="293613d9cc10213d95d2f25301745f2d1774af9b37413c5c9b48b7f7118d8e5c" Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.282811 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.282831 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" containerName="cleanup" Nov 03 20:45:29.290531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.283117 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" containerName="cleanup" Nov 03 20:45:29.298535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.297778 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.302853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.302798 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk" event={"ID":"d43955ae-3dcb-4b1b-9c57-ab375721ed29","Type":"ContainerDied","Data":"c877c7a44cf73c03ed2fe00b3aebbe24ed50473db70ec42926ef5e41c2c52fe7"} Nov 03 20:45:29.303116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.303088 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64"] Nov 03 20:45:29.303275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.303258 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl"] Nov 03 20:45:29.303407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.303387 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"agc-cred-61287f81d6d5\"" Nov 03 20:45:29.303615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.303598 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.304258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.304169 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895645-6brvl"] Nov 03 20:45:29.305167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.305128 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6"] Nov 03 20:45:29.306753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.306717 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"agc-cred-3540e3e4c282\"" Nov 03 20:45:29.317662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.317637 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895645-2sct6"] Nov 03 20:45:29.323096 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.323063 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g"] Nov 03 20:45:29.328983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.328903 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895645-xzz5g"] Nov 03 20:45:29.335030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.335009 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk"] Nov 03 20:45:29.339978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.339962 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d4fe2-29895645-nztjk"] Nov 03 20:45:29.390103 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:29.390059 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43955ae_3dcb_4b1b_9c57_ab375721ed29.slice/crio-c877c7a44cf73c03ed2fe00b3aebbe24ed50473db70ec42926ef5e41c2c52fe7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9feecd9_d5d6_467c_8164_f8f9e6ba7b03.slice/crio-48314be9d14fd1065ad43b5dd3b0d7cf01667ba2397fbd5874da43743490298e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9feecd9_d5d6_467c_8164_f8f9e6ba7b03.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43955ae_3dcb_4b1b_9c57_ab375721ed29.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:45:29.414040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.414014 5815 scope.go:122] "RemoveContainer" containerID="a02b22491efa394cd7b27996e16ff1d825e20fd64201cdf29298e69f74bfc039" Nov 03 20:45:29.415202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.415175 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.415354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.415340 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjp9\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.415647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.415627 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.416055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.416035 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.416296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.416280 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.416485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.416466 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pks8s\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.416670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.416640 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.416792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.416740 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.518359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518267 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.518578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518421 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.518578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518471 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pks8s\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.518578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518541 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.518861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518604 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.518861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518649 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.518861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518674 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjp9\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.518861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.518754 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.521391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.521363 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.521702 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.521678 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.521793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.521697 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.521793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.521780 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.521986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.521965 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.522112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.522097 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.531261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.531241 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjp9\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:29.531697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.531678 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pks8s\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s\") pod \"payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.727721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.727686 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:29.741390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:29.741362 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:30.042718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.042682 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:30.056098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.056060 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw"] Nov 03 20:45:30.060343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.060242 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:30.063671 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:30.063645 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda497eb34_2be2_44df_98c4_9b74da04329f.slice/crio-c1e84e3b67b4ee62f80316941f2d715306b3ef2e66860011713c26c4e024959b WatchSource:0}: Error finding container c1e84e3b67b4ee62f80316941f2d715306b3ef2e66860011713c26c4e024959b: Status 404 returned error can't find the container with id c1e84e3b67b4ee62f80316941f2d715306b3ef2e66860011713c26c4e024959b Nov 03 20:45:30.121681 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:30.121656 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b4bc6d_436c_4573_a82a_548f0fc55e48.slice/crio-29f2cac0a8d189622ebe9691f575e2bc231bd45b05b3ff802778dc6952ace1c4 WatchSource:0}: Error finding container 29f2cac0a8d189622ebe9691f575e2bc231bd45b05b3ff802778dc6952ace1c4: Status 404 returned error can't find the container with id 29f2cac0a8d189622ebe9691f575e2bc231bd45b05b3ff802778dc6952ace1c4 Nov 03 20:45:30.124160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.124134 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64"] Nov 03 20:45:30.459579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.451677 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" event={"ID":"50b4bc6d-436c-4573-a82a-548f0fc55e48","Type":"ContainerStarted","Data":"29f2cac0a8d189622ebe9691f575e2bc231bd45b05b3ff802778dc6952ace1c4"} Nov 03 20:45:30.506069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.506024 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" event={"ID":"ae379051-3d3d-4dc5-8f68-79dba06cb76f","Type":"ContainerStarted","Data":"5f597e836513d6d71961dd59e4021522a6c516284590fd72b923a39eb447eecd"} Nov 03 20:45:30.512159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.506373 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" event={"ID":"ae379051-3d3d-4dc5-8f68-79dba06cb76f","Type":"ContainerStarted","Data":"8fb2f0d51c8913db6b2a553001691ff8e3f4be511eb239c9995e8d518e4502cf"} Nov 03 20:45:30.688138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.688067 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" podStartSLOduration=4.688046006 podStartE2EDuration="4.688046006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:30.643922455 +0000 UTC m=+2640.174268128" watchObservedRunningTime="2026-11-03 20:45:30.688046006 +0000 UTC m=+2640.218391683" Nov 03 20:45:30.832080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.832053 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0264611c-c245-45a0-bef1-db5ae3b23fdf" path="/var/lib/kubelet/pods/0264611c-c245-45a0-bef1-db5ae3b23fdf/volumes" Nov 03 20:45:30.832965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.832949 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9feecd9-d5d6-467c-8164-f8f9e6ba7b03" path="/var/lib/kubelet/pods/a9feecd9-d5d6-467c-8164-f8f9e6ba7b03/volumes" Nov 03 20:45:30.833793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.833779 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43955ae-3dcb-4b1b-9c57-ab375721ed29" path="/var/lib/kubelet/pods/d43955ae-3dcb-4b1b-9c57-ab375721ed29/volumes" Nov 03 20:45:30.834709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.834695 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea08089a-857b-4432-8549-50ca4bf1dbbc" path="/var/lib/kubelet/pods/ea08089a-857b-4432-8549-50ca4bf1dbbc/volumes" Nov 03 20:45:30.835374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.835357 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" event={"ID":"a497eb34-2be2-44df-98c4-9b74da04329f","Type":"ContainerStarted","Data":"c1e84e3b67b4ee62f80316941f2d715306b3ef2e66860011713c26c4e024959b"} Nov 03 20:45:30.842139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:30.842120 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:45:31.846882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.846845 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" event={"ID":"50b4bc6d-436c-4573-a82a-548f0fc55e48","Type":"ContainerStarted","Data":"5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe"} Nov 03 20:45:31.847330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.847008 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:31.849466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.849431 5815 generic.go:356] "Generic (PLEG): container finished" podID="ae379051-3d3d-4dc5-8f68-79dba06cb76f" containerID="5f597e836513d6d71961dd59e4021522a6c516284590fd72b923a39eb447eecd" exitCode=0 Nov 03 20:45:31.849617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.849542 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" event={"ID":"ae379051-3d3d-4dc5-8f68-79dba06cb76f","Type":"ContainerDied","Data":"5f597e836513d6d71961dd59e4021522a6c516284590fd72b923a39eb447eecd"} Nov 03 20:45:31.851867 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.851832 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" event={"ID":"a497eb34-2be2-44df-98c4-9b74da04329f","Type":"ContainerStarted","Data":"8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde"} Nov 03 20:45:31.851976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.851930 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:31.869247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.869161 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" podStartSLOduration=3.869150028 podStartE2EDuration="3.869150028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:31.868127847 +0000 UTC m=+2641.398473543" watchObservedRunningTime="2026-11-03 20:45:31.869150028 +0000 UTC m=+2641.399495700" Nov 03 20:45:31.912203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:31.912157 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" podStartSLOduration=3.912143824 podStartE2EDuration="3.912143824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:31.907472443 +0000 UTC m=+2641.437818116" watchObservedRunningTime="2026-11-03 20:45:31.912143824 +0000 UTC m=+2641.442489496" Nov 03 20:45:33.128920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.128847 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:33.269603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.265136 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca\") pod \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " Nov 03 20:45:33.269603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.265285 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca\") pod \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " Nov 03 20:45:33.269603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.265627 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc\" (UniqueName: \"kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc\") pod \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\" (UID: \"ae379051-3d3d-4dc5-8f68-79dba06cb76f\") " Nov 03 20:45:33.270649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.270615 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca" pod "ae379051-3d3d-4dc5-8f68-79dba06cb76f" (UID: "ae379051-3d3d-4dc5-8f68-79dba06cb76f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:33.287101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.287068 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca" pod "ae379051-3d3d-4dc5-8f68-79dba06cb76f" (UID: "ae379051-3d3d-4dc5-8f68-79dba06cb76f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:33.287603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.287558 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc" pod "ae379051-3d3d-4dc5-8f68-79dba06cb76f" (UID: "ae379051-3d3d-4dc5-8f68-79dba06cb76f"). InnerVolumeSpecName "kube-api-access-8wbvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:33.368375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.368340 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:33.368375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.368373 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae379051-3d3d-4dc5-8f68-79dba06cb76f-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:33.368375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.368383 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbvc\" (UniqueName: \"kubernetes.io/projected/ae379051-3d3d-4dc5-8f68-79dba06cb76f-kube-api-access-8wbvc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:33.944267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.944235 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" Nov 03 20:45:33.944465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.944345 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg" event={"ID":"ae379051-3d3d-4dc5-8f68-79dba06cb76f","Type":"ContainerDied","Data":"8fb2f0d51c8913db6b2a553001691ff8e3f4be511eb239c9995e8d518e4502cf"} Nov 03 20:45:33.944465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:33.944397 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fb2f0d51c8913db6b2a553001691ff8e3f4be511eb239c9995e8d518e4502cf" Nov 03 20:45:34.328445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.327460 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr"] Nov 03 20:45:34.335354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.335316 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ae379051-3d3d-4dc5-8f68-79dba06cb76f" containerName="revoke-keys" Nov 03 20:45:34.335354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.335354 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae379051-3d3d-4dc5-8f68-79dba06cb76f" containerName="revoke-keys" Nov 03 20:45:34.335618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.335606 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae379051-3d3d-4dc5-8f68-79dba06cb76f" containerName="revoke-keys" Nov 03 20:45:34.340346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.340321 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr"] Nov 03 20:45:34.340746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.340710 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.348763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.348738 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c94fdf\"/\"kube-root-ca.crt\"" Nov 03 20:45:34.348901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.348826 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c94fdf\"/\"openshift-service-ca.crt\"" Nov 03 20:45:34.349396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.349377 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c94fdf\"/\"default-dockercfg-grp6k\"" Nov 03 20:45:34.349644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.349627 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c94fdf\"/\"test-model-c94fdf-kserve-self-signed-certs\"" Nov 03 20:45:34.547553 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.547497 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.548069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.548038 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.567115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.567079 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.567458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.567418 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.567620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.567602 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7jt\" (UniqueName: \"kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.568447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.568418 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.712528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.712469 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.719574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.719523 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.719955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.719920 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.720188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.720157 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.720356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.720324 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7jt\" (UniqueName: \"kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.720552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.720496 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.721977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.721946 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.738062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.723757 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.742352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.740726 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.763753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.763694 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.775276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.775239 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7jt\" (UniqueName: \"kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.814575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.786445 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs\") pod \"test-model-c94fdf-kserve-54876c4795-gsmnr\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:34.990739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:34.990643 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:35.309938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:35.309906 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr"] Nov 03 20:45:35.310382 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:35.310356 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf209fe5c_d690_4126_86f4_d599428d512f.slice/crio-580d19f8d374fb59c980db14456bd28c749498d40f2a36c709a36037a9839f4e WatchSource:0}: Error finding container 580d19f8d374fb59c980db14456bd28c749498d40f2a36c709a36037a9839f4e: Status 404 returned error can't find the container with id 580d19f8d374fb59c980db14456bd28c749498d40f2a36c709a36037a9839f4e Nov 03 20:45:36.017097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.017061 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" event={"ID":"f209fe5c-d690-4126-86f4-d599428d512f","Type":"ContainerStarted","Data":"5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c"} Nov 03 20:45:36.017612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.017106 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" event={"ID":"f209fe5c-d690-4126-86f4-d599428d512f","Type":"ContainerStarted","Data":"580d19f8d374fb59c980db14456bd28c749498d40f2a36c709a36037a9839f4e"} Nov 03 20:45:36.041426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.041379 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podStartSLOduration=2.041365778 podStartE2EDuration="2.041365778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:36.036413306 +0000 UTC m=+2645.566758980" watchObservedRunningTime="2026-11-03 20:45:36.041365778 +0000 UTC m=+2645.571711450" Nov 03 20:45:36.838866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.838832 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b"] Nov 03 20:45:36.855329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.855301 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b"] Nov 03 20:45:36.855552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.855412 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:36.875601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.871042 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8c6934\"/\"kube-root-ca.crt\"" Nov 03 20:45:36.875601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.871042 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8c6934\"/\"default-dockercfg-26wvm\"" Nov 03 20:45:36.875601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.871110 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8c6934\"/\"test-model-8c6934-kserve-self-signed-certs\"" Nov 03 20:45:36.875601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.871388 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8c6934\"/\"openshift-service-ca.crt\"" Nov 03 20:45:36.991587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.991558 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:36.991789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.991604 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:36.991789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.991636 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zcs\" (UniqueName: \"kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:36.991961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.991910 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:36.992099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.992084 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:36.992183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.992169 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:36.992272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:36.992250 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.093931 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.094043 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.094084 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.094214 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.094248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zcs\" (UniqueName: \"kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.094482 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.097193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.095057 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.098846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.098817 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.100209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.100187 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.114558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.112600 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.143586 5815 generic.go:356] "Generic (PLEG): container finished" podID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerID="4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1" exitCode=0 Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.147585 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" event={"ID":"0c95d6d4-ccfb-4178-a3cb-72700e6de96d","Type":"ContainerDied","Data":"4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1"} Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.148067 5815 scope.go:122] "RemoveContainer" containerID="4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1" Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.172231 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.172326 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.172374 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:45:37.187022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.175252 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zcs\" (UniqueName: \"kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.251139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.246846 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm\") pod \"test-model-8c6934-kserve-66cdbd5bdf-wzt5b\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:37.495942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:37.495741 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:38.399372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.399276 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b"] Nov 03 20:45:38.427688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.420152 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k"] Nov 03 20:45:38.439921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.439896 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k"] Nov 03 20:45:38.440213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.440196 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.452795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.452769 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:45:38.456298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.453176 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:45:38.458640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.458608 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" event={"ID":"0c95d6d4-ccfb-4178-a3cb-72700e6de96d","Type":"ContainerStarted","Data":"e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640"} Nov 03 20:45:38.611039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.611006 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.611377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.611128 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2gj\" (UniqueName: \"kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.611377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.611192 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.712896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.712860 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.713209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.713189 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2gj\" (UniqueName: \"kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.713312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.713256 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.714037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.714008 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.715570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.715551 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.725331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.725309 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2gj\" (UniqueName: \"kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj\") pod \"maas-api-revoke-sub-e58d5ff0-jzr5k\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:38.787474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:38.787447 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:39.127579 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:39.121034 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd7d3eb_fca3_4858_97ce_f6aafc487f25.slice/crio-9ba523fe52ee7d221e3273b937b9174ba82153083ee34b6fc49bd9ac235f845f WatchSource:0}: Error finding container 9ba523fe52ee7d221e3273b937b9174ba82153083ee34b6fc49bd9ac235f845f: Status 404 returned error can't find the container with id 9ba523fe52ee7d221e3273b937b9174ba82153083ee34b6fc49bd9ac235f845f Nov 03 20:45:39.127579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.127466 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k"] Nov 03 20:45:39.156114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.156090 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:45:39.156425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.156402 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:45:39.156552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.156454 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:45:39.511348 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.511302 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" event={"ID":"efd7d3eb-fca3-4858-97ce-f6aafc487f25","Type":"ContainerStarted","Data":"9ba523fe52ee7d221e3273b937b9174ba82153083ee34b6fc49bd9ac235f845f"} Nov 03 20:45:39.515618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.515588 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" event={"ID":"088ab3f6-0447-48d9-ac71-bfaa1b8f2791","Type":"ContainerStarted","Data":"8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631"} Nov 03 20:45:39.515783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.515628 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" event={"ID":"088ab3f6-0447-48d9-ac71-bfaa1b8f2791","Type":"ContainerStarted","Data":"32f9819284bec3befc5ba2a5d30b2154befb637a1c50c44986dca5b2cfe410b2"} Nov 03 20:45:39.540657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:39.540610 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podStartSLOduration=3.540597188 podStartE2EDuration="3.540597188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:39.535720601 +0000 UTC m=+2649.066066274" watchObservedRunningTime="2026-11-03 20:45:39.540597188 +0000 UTC m=+2649.070942860" Nov 03 20:45:40.155905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:40.155823 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:45:40.155905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:40.155888 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:45:40.498881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:40.498812 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:40.530627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:40.530598 5815 generic.go:356] "Generic (PLEG): container finished" podID="efd7d3eb-fca3-4858-97ce-f6aafc487f25" containerID="91950b99b3ba8463e160947994c84a7e2f88d92928ba23da0eeafd28a25cf26c" exitCode=0 Nov 03 20:45:40.531036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:40.530662 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" event={"ID":"efd7d3eb-fca3-4858-97ce-f6aafc487f25","Type":"ContainerDied","Data":"91950b99b3ba8463e160947994c84a7e2f88d92928ba23da0eeafd28a25cf26c"} Nov 03 20:45:41.026429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.026389 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc"] Nov 03 20:45:41.031463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.031435 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682c78a3-87frc"] Nov 03 20:45:41.156372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.156339 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:45:41.156570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.156394 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:45:41.850349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.850322 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:41.946001 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.945845 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca\") pod \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " Nov 03 20:45:41.946901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.946875 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj\" (UniqueName: \"kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj\") pod \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " Nov 03 20:45:41.947030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.946955 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca\") pod \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\" (UID: \"efd7d3eb-fca3-4858-97ce-f6aafc487f25\") " Nov 03 20:45:41.948212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.948177 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca" pod "efd7d3eb-fca3-4858-97ce-f6aafc487f25" (UID: "efd7d3eb-fca3-4858-97ce-f6aafc487f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:41.949327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.949298 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca" pod "efd7d3eb-fca3-4858-97ce-f6aafc487f25" (UID: "efd7d3eb-fca3-4858-97ce-f6aafc487f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:41.953481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.952827 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:41.953481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.952871 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd7d3eb-fca3-4858-97ce-f6aafc487f25-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:41.955383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:41.954944 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj" pod "efd7d3eb-fca3-4858-97ce-f6aafc487f25" (UID: "efd7d3eb-fca3-4858-97ce-f6aafc487f25"). InnerVolumeSpecName "kube-api-access-rd2gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:42.055596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.055560 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2gj\" (UniqueName: \"kubernetes.io/projected/efd7d3eb-fca3-4858-97ce-f6aafc487f25-kube-api-access-rd2gj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:42.156246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.156159 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:45:42.156396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.156238 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:45:42.556328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.556290 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" event={"ID":"efd7d3eb-fca3-4858-97ce-f6aafc487f25","Type":"ContainerDied","Data":"9ba523fe52ee7d221e3273b937b9174ba82153083ee34b6fc49bd9ac235f845f"} Nov 03 20:45:42.556550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.556348 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ba523fe52ee7d221e3273b937b9174ba82153083ee34b6fc49bd9ac235f845f" Nov 03 20:45:42.556550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.556305 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k" Nov 03 20:45:42.613043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.613012 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5" path="/var/lib/kubelet/pods/3e68fb13-e7a9-435a-a3c3-a403a2e3a1a5/volumes" Nov 03 20:45:42.897577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.897287 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:45:42.898083 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.898042 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:45:42.956023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:42.955981 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2"] Nov 03 20:45:43.018665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:43.018631 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9"] Nov 03 20:45:43.196814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:43.196674 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:45:43.197105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:43.197071 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:45:43.853924 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:43.853890 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:45:44.156718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:44.156639 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body= Nov 03 20:45:44.156718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:44.156695 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" Nov 03 20:45:44.975091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:44.975057 5815 scope.go:122] "RemoveContainer" containerID="a86886beadb89b143b270d876690a00b496b6bde5dc8b7b29404aafe41d9f34d" Nov 03 20:45:44.991868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:44.991040 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:45.007144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.007114 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:45.239540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.238745 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:45:45.239540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.239419 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:45:45.241311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.241269 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:45:45.631606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.628992 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp"] Nov 03 20:45:45.631606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.630331 5815 cpu_manager.go:414] "containerMap: removing container" podUID="efd7d3eb-fca3-4858-97ce-f6aafc487f25" containerName="revoke-keys" Nov 03 20:45:45.631606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.630350 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd7d3eb-fca3-4858-97ce-f6aafc487f25" containerName="revoke-keys" Nov 03 20:45:45.631606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.630538 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd7d3eb-fca3-4858-97ce-f6aafc487f25" containerName="revoke-keys" Nov 03 20:45:45.643385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.641933 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.643385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.642194 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp"] Nov 03 20:45:45.649166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.648384 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f5afd7\"/\"kube-root-ca.crt\"" Nov 03 20:45:45.649166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.649152 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f5afd7\"/\"test-model-f5afd7-kserve-self-signed-certs\"" Nov 03 20:45:45.654722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.649487 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f5afd7\"/\"default-dockercfg-jv88r\"" Nov 03 20:45:45.690592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.675805 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:45:45.693376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.692805 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.693376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.693043 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.693376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.693134 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.693376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.693199 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbgx\" (UniqueName: \"kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.693376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.693247 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.693376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.693282 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.701947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.701880 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f5afd7\"/\"openshift-service-ca.crt\"" Nov 03 20:45:45.795958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.795913 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.796219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.796151 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.796219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.796211 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.796372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.796246 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbgx\" (UniqueName: \"kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.796372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.796300 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.796372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.796327 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.797529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.796796 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.798415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.798395 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.798784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.798766 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.800414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.800297 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.800784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.800759 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.813582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.809798 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbgx\" (UniqueName: \"kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx\") pod \"test-model-f5afd7-kserve-76fdd9f757-wd9hp\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:45.989533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:45.976917 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:46.449599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:46.447465 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp"] Nov 03 20:45:46.452037 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:45:46.451928 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d409156_98f0_4fcf_9147_b581d5610d17.slice/crio-39d162e4ca1c95aae3a094afc018bcc6e9df6fac1d3de845f1977f4b43fd96a0 WatchSource:0}: Error finding container 39d162e4ca1c95aae3a094afc018bcc6e9df6fac1d3de845f1977f4b43fd96a0: Status 404 returned error can't find the container with id 39d162e4ca1c95aae3a094afc018bcc6e9df6fac1d3de845f1977f4b43fd96a0 Nov 03 20:45:46.667191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:46.667150 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" event={"ID":"2d409156-98f0-4fcf-9147-b581d5610d17","Type":"ContainerStarted","Data":"af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0"} Nov 03 20:45:46.667375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:46.667202 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" event={"ID":"2d409156-98f0-4fcf-9147-b581d5610d17","Type":"ContainerStarted","Data":"39d162e4ca1c95aae3a094afc018bcc6e9df6fac1d3de845f1977f4b43fd96a0"} Nov 03 20:45:46.696460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:46.696362 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podStartSLOduration=1.696343707 podStartE2EDuration="1.696343707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:46.686609181 +0000 UTC m=+2656.216954854" watchObservedRunningTime="2026-11-03 20:45:46.696343707 +0000 UTC m=+2656.226689384" Nov 03 20:45:46.980191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:46.980164 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:47.496455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:47.496420 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:47.510244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:47.510218 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:47.692404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:47.692371 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:45:47.956986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:47.956876 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" podUID="0dae5afa-11a2-482d-8f16-7949dd157d58" containerName="payload-processing" containerID="cri-o://fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b" gracePeriod=25 Nov 03 20:45:48.019667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.019632 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" podUID="2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" containerName="payload-processing" containerID="cri-o://a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45" gracePeriod=25 Nov 03 20:45:48.401223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.401194 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:45:48.403820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.403798 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:45:48.495539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.495487 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj\") pod \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " Nov 03 20:45:48.495723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.495583 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume\") pod \"0dae5afa-11a2-482d-8f16-7949dd157d58\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " Nov 03 20:45:48.495723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.495622 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume\") pod \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " Nov 03 20:45:48.495880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.495829 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume\") pod \"0dae5afa-11a2-482d-8f16-7949dd157d58\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " Nov 03 20:45:48.495949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.495936 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials\") pod \"0dae5afa-11a2-482d-8f16-7949dd157d58\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " Nov 03 20:45:48.496064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.496045 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials\") pod \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " Nov 03 20:45:48.496247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.496230 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume\") pod \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\" (UID: \"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f\") " Nov 03 20:45:48.496331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.496263 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf\") pod \"0dae5afa-11a2-482d-8f16-7949dd157d58\" (UID: \"0dae5afa-11a2-482d-8f16-7949dd157d58\") " Nov 03 20:45:48.498151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.498096 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume" pod "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" (UID: "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:48.498667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.498480 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume" pod "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" (UID: "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:48.499179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.499152 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume" pod "0dae5afa-11a2-482d-8f16-7949dd157d58" (UID: "0dae5afa-11a2-482d-8f16-7949dd157d58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:48.499370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.499342 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume" pod "0dae5afa-11a2-482d-8f16-7949dd157d58" (UID: "0dae5afa-11a2-482d-8f16-7949dd157d58"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:48.510252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.510146 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf" pod "0dae5afa-11a2-482d-8f16-7949dd157d58" (UID: "0dae5afa-11a2-482d-8f16-7949dd157d58"). InnerVolumeSpecName "kube-api-access-hghvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:48.510252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.510191 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials" pod "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" (UID: "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:48.511650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.511613 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj" pod "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" (UID: "2cb3bac6-6e00-4fba-aa9c-b4d3146d118f"). InnerVolumeSpecName "kube-api-access-v2wlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:48.512619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.512593 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials" pod "0dae5afa-11a2-482d-8f16-7949dd157d58" (UID: "0dae5afa-11a2-482d-8f16-7949dd157d58"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:48.598360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598328 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598358 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hghvf\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-kube-api-access-hghvf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598374 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2wlj\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-kube-api-access-v2wlj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598387 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598400 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598412 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dae5afa-11a2-482d-8f16-7949dd157d58-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598424 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0dae5afa-11a2-482d-8f16-7949dd157d58-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.598611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.598438 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:48.725447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.722006 5815 generic.go:356] "Generic (PLEG): container finished" podID="2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" containerID="a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45" exitCode=0 Nov 03 20:45:48.725447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.722199 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" event={"ID":"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f","Type":"ContainerDied","Data":"a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45"} Nov 03 20:45:48.725447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.722254 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" event={"ID":"2cb3bac6-6e00-4fba-aa9c-b4d3146d118f","Type":"ContainerDied","Data":"c6accfcfcebb96e330dae20951a4642d6fdc9b5045223196a1707c27a181c243"} Nov 03 20:45:48.725447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.722284 5815 scope.go:122] "RemoveContainer" containerID="a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45" Nov 03 20:45:48.725447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.723100 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9" Nov 03 20:45:48.758664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.758212 5815 generic.go:356] "Generic (PLEG): container finished" podID="0dae5afa-11a2-482d-8f16-7949dd157d58" containerID="fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b" exitCode=0 Nov 03 20:45:48.759222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.759073 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" Nov 03 20:45:48.759589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.759520 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" event={"ID":"0dae5afa-11a2-482d-8f16-7949dd157d58","Type":"ContainerDied","Data":"fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b"} Nov 03 20:45:48.759589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.759573 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2" event={"ID":"0dae5afa-11a2-482d-8f16-7949dd157d58","Type":"ContainerDied","Data":"b0ce5b19f6c86123a5a01c95f50076b74edcd09c33a94c78274a6772edaff577"} Nov 03 20:45:48.764426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.764407 5815 scope.go:122] "RemoveContainer" containerID="a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45" Nov 03 20:45:48.764776 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:48.764746 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45\": container with ID starting with a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45 not found: ID does not exist" containerID="a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45" Nov 03 20:45:48.764823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.764789 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45"} err="failed to get container status \"a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45\": rpc error: code = NotFound desc = could not find container \"a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45\": container with ID starting with a9f4593ef4eaf9d46a9c228e0782334bfdc453b001fc5d6bdea1f2bebb8aef45 not found: ID does not exist" Nov 03 20:45:48.764823 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.764813 5815 scope.go:122] "RemoveContainer" containerID="fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b" Nov 03 20:45:48.782657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.782607 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9"] Nov 03 20:45:48.786673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.786642 5815 scope.go:122] "RemoveContainer" containerID="fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b" Nov 03 20:45:48.787024 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:45:48.786998 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b\": container with ID starting with fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b not found: ID does not exist" containerID="fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b" Nov 03 20:45:48.787119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.787035 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b"} err="failed to get container status \"fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b\": rpc error: code = NotFound desc = could not find container \"fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b\": container with ID starting with fcb5c985412998fe1490902d668b083bea12f0621804061858b5fd01e2e9848b not found: ID does not exist" Nov 03 20:45:48.792226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.792205 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbbczk9"] Nov 03 20:45:48.800415 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.800394 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2"] Nov 03 20:45:48.805793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:48.805772 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfqrmt2"] Nov 03 20:45:50.600532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:50.600389 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dae5afa-11a2-482d-8f16-7949dd157d58" path="/var/lib/kubelet/pods/0dae5afa-11a2-482d-8f16-7949dd157d58/volumes" Nov 03 20:45:50.605825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:50.601412 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" path="/var/lib/kubelet/pods/2cb3bac6-6e00-4fba-aa9c-b4d3146d118f/volumes" Nov 03 20:45:52.043534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:52.039955 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj"] Nov 03 20:45:52.047793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:52.047760 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aba4f5b-gmggj"] Nov 03 20:45:52.589938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:52.589904 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0" path="/var/lib/kubelet/pods/4a43b36f-d4c9-46b9-a0fc-0edcf3a37be0/volumes" Nov 03 20:45:53.027125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:53.027084 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l"] Nov 03 20:45:53.032165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:53.032134 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e370b2-qvh7l"] Nov 03 20:45:54.589634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:54.589599 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaae1739-220c-4432-a050-685c13cd1f30" path="/var/lib/kubelet/pods/eaae1739-220c-4432-a050-685c13cd1f30/volumes" Nov 03 20:45:55.979434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:55.979401 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:55.992788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:55.992760 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:56.145379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:56.145350 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:45:57.050573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:57.049306 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd"] Nov 03 20:45:57.054834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:57.054658 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f981cd5-9fghd"] Nov 03 20:45:58.026581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:58.026549 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s"] Nov 03 20:45:58.031596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:58.031569 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99edc7a-dzj5s"] Nov 03 20:45:58.600135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:58.600094 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870" path="/var/lib/kubelet/pods/1b61b1ff-d2fc-4ff0-96fd-cc67dc0db870/volumes" Nov 03 20:45:58.601126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:58.601101 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f28329-b52f-4b55-9d92-f9e59f268c22" path="/var/lib/kubelet/pods/c1f28329-b52f-4b55-9d92-f9e59f268c22/volumes" Nov 03 20:45:59.902532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:45:59.902373 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" probeResult="failure" output="Get \"https://10.132.1.110:8000/ready\": context deadline exceeded" Nov 03 20:46:08.055533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:08.054625 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk"] Nov 03 20:46:08.075555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:08.075286 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bfc53bb-xw6vk"] Nov 03 20:46:08.595567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:08.595474 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64" path="/var/lib/kubelet/pods/3f5ca1de-9cf6-4e5c-b52b-cda4585c8f64/volumes" Nov 03 20:46:13.853314 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:46:13.853275 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:46:16.029170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:16.029133 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf"] Nov 03 20:46:16.035893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:16.035866 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-915549f5-jpncf"] Nov 03 20:46:16.590038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:16.590004 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b2dc00-9f61-4bfe-8775-86cad0d4fee7" path="/var/lib/kubelet/pods/d0b2dc00-9f61-4bfe-8775-86cad0d4fee7/volumes" Nov 03 20:46:17.364537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:17.358265 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:46:19.067601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:19.066585 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b"] Nov 03 20:46:19.106534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:19.102062 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc87b8c-h9f7b"] Nov 03 20:46:19.692528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:19.690500 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" probeResult="failure" output="Get \"https://10.132.1.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:19.720236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:19.719648 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:19.739200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:19.738637 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:19.845058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:19.844990 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" probeResult="failure" output="Get \"https://10.132.1.110:8000/ready\": context deadline exceeded" Nov 03 20:46:20.058993 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:20.053417 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:20.171815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:20.148063 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" probeResult="failure" output="Get \"https://10.132.1.117:8000/ready\": context deadline exceeded" Nov 03 20:46:20.447586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:20.419959 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:20.447586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:20.439816 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Nov 03 20:46:20.456767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:20.455462 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:20.632665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:20.632628 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95422424-c192-400e-8c4c-40b48c4a9d25" path="/var/lib/kubelet/pods/95422424-c192-400e-8c4c-40b48c4a9d25/volumes" Nov 03 20:46:23.148529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.148486 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg"] Nov 03 20:46:23.149389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.149371 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" containerName="payload-processing" Nov 03 20:46:23.149389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.149390 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" containerName="payload-processing" Nov 03 20:46:23.149545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.149419 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0dae5afa-11a2-482d-8f16-7949dd157d58" containerName="payload-processing" Nov 03 20:46:23.149545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.149424 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dae5afa-11a2-482d-8f16-7949dd157d58" containerName="payload-processing" Nov 03 20:46:23.149640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.149628 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb3bac6-6e00-4fba-aa9c-b4d3146d118f" containerName="payload-processing" Nov 03 20:46:23.149679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.149648 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dae5afa-11a2-482d-8f16-7949dd157d58" containerName="payload-processing" Nov 03 20:46:23.155957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.155934 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.160837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.160661 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:46:23.161545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.161522 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg"] Nov 03 20:46:23.161713 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.161691 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:46:23.327452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.327413 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.327652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.327581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft2n5\" (UniqueName: \"kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.327733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.327712 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.429740 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.429663 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.429894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.429761 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft2n5\" (UniqueName: \"kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.429894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.429838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.430584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.430560 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.432644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.432615 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.442216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.442193 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft2n5\" (UniqueName: \"kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5\") pod \"maas-api-revoke-sub-cb9ae211-8mtwg\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.478791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.478760 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:23.793379 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:46:23.793341 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea10618_9026_4a7c_8edc_07f6863a8f89.slice/crio-dac40491598a836ad43058722d1e3054633849e16180a7763d146ab0ab2338b4 WatchSource:0}: Error finding container dac40491598a836ad43058722d1e3054633849e16180a7763d146ab0ab2338b4: Status 404 returned error can't find the container with id dac40491598a836ad43058722d1e3054633849e16180a7763d146ab0ab2338b4 Nov 03 20:46:23.795084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:23.795055 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg"] Nov 03 20:46:24.333489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:24.333445 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" event={"ID":"aea10618-9026-4a7c-8edc-07f6863a8f89","Type":"ContainerStarted","Data":"dac40491598a836ad43058722d1e3054633849e16180a7763d146ab0ab2338b4"} Nov 03 20:46:25.344626 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:25.344594 5815 generic.go:356] "Generic (PLEG): container finished" podID="aea10618-9026-4a7c-8edc-07f6863a8f89" containerID="fe21b5b97205e6b1c0dc47400863db6511d9b3dc8cf7491ff9590d510f12973b" exitCode=0 Nov 03 20:46:25.345113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:25.344649 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" event={"ID":"aea10618-9026-4a7c-8edc-07f6863a8f89","Type":"ContainerDied","Data":"fe21b5b97205e6b1c0dc47400863db6511d9b3dc8cf7491ff9590d510f12973b"} Nov 03 20:46:26.622499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.622474 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:26.705100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.705016 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca\") pod \"aea10618-9026-4a7c-8edc-07f6863a8f89\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " Nov 03 20:46:26.705280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.705139 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca\") pod \"aea10618-9026-4a7c-8edc-07f6863a8f89\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " Nov 03 20:46:26.705280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.705207 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5\" (UniqueName: \"kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5\") pod \"aea10618-9026-4a7c-8edc-07f6863a8f89\" (UID: \"aea10618-9026-4a7c-8edc-07f6863a8f89\") " Nov 03 20:46:26.706843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.706811 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca" pod "aea10618-9026-4a7c-8edc-07f6863a8f89" (UID: "aea10618-9026-4a7c-8edc-07f6863a8f89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:26.708643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.708613 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca" pod "aea10618-9026-4a7c-8edc-07f6863a8f89" (UID: "aea10618-9026-4a7c-8edc-07f6863a8f89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:26.708827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.708808 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5" pod "aea10618-9026-4a7c-8edc-07f6863a8f89" (UID: "aea10618-9026-4a7c-8edc-07f6863a8f89"). InnerVolumeSpecName "kube-api-access-ft2n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:26.807814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.807778 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft2n5\" (UniqueName: \"kubernetes.io/projected/aea10618-9026-4a7c-8edc-07f6863a8f89-kube-api-access-ft2n5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.807814 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.807814 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:26.808097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:26.807828 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea10618-9026-4a7c-8edc-07f6863a8f89-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:27.404312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:27.404282 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" Nov 03 20:46:27.404680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:27.404323 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg" event={"ID":"aea10618-9026-4a7c-8edc-07f6863a8f89","Type":"ContainerDied","Data":"dac40491598a836ad43058722d1e3054633849e16180a7763d146ab0ab2338b4"} Nov 03 20:46:27.404680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:27.404375 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac40491598a836ad43058722d1e3054633849e16180a7763d146ab0ab2338b4" Nov 03 20:46:32.028550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:32.028516 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg"] Nov 03 20:46:32.033276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:32.033254 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0b355e-8w9rg"] Nov 03 20:46:32.590066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:32.590031 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90da652a-1f80-4e01-bf1d-7ef8a5f004ea" path="/var/lib/kubelet/pods/90da652a-1f80-4e01-bf1d-7ef8a5f004ea/volumes" Nov 03 20:46:33.364951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.364916 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9"] Nov 03 20:46:33.366266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.366244 5815 cpu_manager.go:414] "containerMap: removing container" podUID="aea10618-9026-4a7c-8edc-07f6863a8f89" containerName="revoke-keys" Nov 03 20:46:33.366344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.366270 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea10618-9026-4a7c-8edc-07f6863a8f89" containerName="revoke-keys" Nov 03 20:46:33.366484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.366472 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea10618-9026-4a7c-8edc-07f6863a8f89" containerName="revoke-keys" Nov 03 20:46:33.372349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.372325 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.376472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.376449 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-db5198-openshift-default-dockercfg-wv2ss\"" Nov 03 20:46:33.381994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.381970 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9"] Nov 03 20:46:33.467680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.467640 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.467862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.467709 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.467862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.467832 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.467968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.467910 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pkl\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.468094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.468060 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.468213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.468196 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.468279 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.468261 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.468383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.468366 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.468443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.468427 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570172 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pkl\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570247 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570417 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570489 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570588 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570683 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570736 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570685 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.570904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.571057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.570933 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.571057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.571020 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.571190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.571093 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.571190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.571101 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.571320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.571209 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.571380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.571328 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.573874 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.573843 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.573999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.573911 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.583075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.583050 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.583659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.583636 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pkl\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.616099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.616039 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225"] Nov 03 20:46:33.616239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.616221 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" containerID="cri-o://d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699" gracePeriod=30 Nov 03 20:46:33.691801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.691768 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:33.786786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.786752 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj"] Nov 03 20:46:33.796820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.796785 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:33.801718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.801111 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:46:33.801718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.801157 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:46:33.805841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.804745 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj"] Nov 03 20:46:33.881091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.880937 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwssd\" (UniqueName: \"kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:33.881091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.881017 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:33.881328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:33.881123 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.018536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.017087 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwssd\" (UniqueName: \"kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.018536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.017170 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.018536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.017211 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.019722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.019492 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.030131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.030100 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.037589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.037560 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwssd\" (UniqueName: \"kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd\") pod \"maas-api-revoke-sub-1a6c3d40-7fzwj\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.053108 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:46:34.053082 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a087036_27f7_4f16_afc1_e2e4bceb248b.slice/crio-95bbb6e1bcfa35d08ed5f52a6c5c6cfac6feea9811c48f5cb28f9f8d78a81e59 WatchSource:0}: Error finding container 95bbb6e1bcfa35d08ed5f52a6c5c6cfac6feea9811c48f5cb28f9f8d78a81e59: Status 404 returned error can't find the container with id 95bbb6e1bcfa35d08ed5f52a6c5c6cfac6feea9811c48f5cb28f9f8d78a81e59 Nov 03 20:46:34.054095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.054074 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9"] Nov 03 20:46:34.056291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.056261 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:46:34.056367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.056324 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:46:34.056367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.056352 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:46:34.143095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.143073 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:34.449536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.449328 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj"] Nov 03 20:46:34.452862 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:46:34.450642 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd5092d_8651_4d95_a794_30c12767a10d.slice/crio-7b8d899b28acf3c5a719b909a90fbadba354f1439541b41bc904579124377b29 WatchSource:0}: Error finding container 7b8d899b28acf3c5a719b909a90fbadba354f1439541b41bc904579124377b29: Status 404 returned error can't find the container with id 7b8d899b28acf3c5a719b909a90fbadba354f1439541b41bc904579124377b29 Nov 03 20:46:34.534335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.534297 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" event={"ID":"9a087036-27f7-4f16-afc1-e2e4bceb248b","Type":"ContainerStarted","Data":"d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc"} Nov 03 20:46:34.534526 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.534362 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" event={"ID":"9a087036-27f7-4f16-afc1-e2e4bceb248b","Type":"ContainerStarted","Data":"95bbb6e1bcfa35d08ed5f52a6c5c6cfac6feea9811c48f5cb28f9f8d78a81e59"} Nov 03 20:46:34.535903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.535871 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" event={"ID":"abd5092d-8651-4d95-a794-30c12767a10d","Type":"ContainerStarted","Data":"7b8d899b28acf3c5a719b909a90fbadba354f1439541b41bc904579124377b29"} Nov 03 20:46:34.559885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.559839 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podStartSLOduration=1.55982522 podStartE2EDuration="1.55982522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:34.556711061 +0000 UTC m=+2704.087056745" watchObservedRunningTime="2026-11-03 20:46:34.55982522 +0000 UTC m=+2704.090170889" Nov 03 20:46:34.692696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:34.692666 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:35.318722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.318698 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:46:35.321753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.321719 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:46:35.325716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.325699 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:46:35.327811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.327793 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:46:35.613574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.601175 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" event={"ID":"abd5092d-8651-4d95-a794-30c12767a10d","Type":"ContainerStarted","Data":"526c62c86685ac619476709f6f391bfca58b902ed6dd72e3095156cc69be840d"} Nov 03 20:46:35.698684 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.698643 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:46:35.698893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:35.698701 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:46:36.614737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:36.614704 5815 generic.go:356] "Generic (PLEG): container finished" podID="abd5092d-8651-4d95-a794-30c12767a10d" containerID="526c62c86685ac619476709f6f391bfca58b902ed6dd72e3095156cc69be840d" exitCode=0 Nov 03 20:46:36.615155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:36.615062 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" event={"ID":"abd5092d-8651-4d95-a794-30c12767a10d","Type":"ContainerDied","Data":"526c62c86685ac619476709f6f391bfca58b902ed6dd72e3095156cc69be840d"} Nov 03 20:46:36.693528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:36.692945 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:46:36.693528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:36.693003 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:46:37.697350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:37.696908 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:46:37.697350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:37.696972 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:46:37.928762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:37.928734 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:38.052664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.048952 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca\") pod \"abd5092d-8651-4d95-a794-30c12767a10d\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " Nov 03 20:46:38.052664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.049342 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd\" (UniqueName: \"kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd\") pod \"abd5092d-8651-4d95-a794-30c12767a10d\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " Nov 03 20:46:38.052664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.049423 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca\") pod \"abd5092d-8651-4d95-a794-30c12767a10d\" (UID: \"abd5092d-8651-4d95-a794-30c12767a10d\") " Nov 03 20:46:38.082783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.081222 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca" pod "abd5092d-8651-4d95-a794-30c12767a10d" (UID: "abd5092d-8651-4d95-a794-30c12767a10d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:38.120975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.115036 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd" pod "abd5092d-8651-4d95-a794-30c12767a10d" (UID: "abd5092d-8651-4d95-a794-30c12767a10d"). InnerVolumeSpecName "kube-api-access-cwssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:38.147739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.142574 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca" pod "abd5092d-8651-4d95-a794-30c12767a10d" (UID: "abd5092d-8651-4d95-a794-30c12767a10d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:38.152904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.152874 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:38.153107 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.153091 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwssd\" (UniqueName: \"kubernetes.io/projected/abd5092d-8651-4d95-a794-30c12767a10d-kube-api-access-cwssd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:38.153215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.153201 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd5092d-8651-4d95-a794-30c12767a10d-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:38.641296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.641267 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" Nov 03 20:46:38.641449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.641339 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj" event={"ID":"abd5092d-8651-4d95-a794-30c12767a10d","Type":"ContainerDied","Data":"7b8d899b28acf3c5a719b909a90fbadba354f1439541b41bc904579124377b29"} Nov 03 20:46:38.641449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.641380 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8d899b28acf3c5a719b909a90fbadba354f1439541b41bc904579124377b29" Nov 03 20:46:38.699011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.698980 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:46:38.699490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:38.699035 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:46:39.145999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.145973 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209391 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209429 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209525 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209582 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209778 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209815 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.209882 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.211283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.210087 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data\") pod \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\" (UID: \"1cb44027-53ca-4bd8-8d11-fb1909ecf87d\") " Nov 03 20:46:39.212034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.212008 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:39.212379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.212359 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:39.213249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.213073 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:39.213784 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.213497 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:39.216985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.215537 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:39.216985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.216271 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "kube-api-access-llgkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:39.226871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.226837 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:39.226976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.226886 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:39.228924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.228902 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo" pod "1cb44027-53ca-4bd8-8d11-fb1909ecf87d" (UID: "1cb44027-53ca-4bd8-8d11-fb1909ecf87d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:39.317257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.317220 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318196 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318223 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318237 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318250 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318265 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgkg\" (UniqueName: \"kubernetes.io/projected/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-kube-api-access-llgkg\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318280 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318293 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.318325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.318306 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb44027-53ca-4bd8-8d11-fb1909ecf87d-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:39.652926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.652835 5815 generic.go:356] "Generic (PLEG): container finished" podID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerID="d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699" exitCode=0 Nov 03 20:46:39.652926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.652898 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" event={"ID":"1cb44027-53ca-4bd8-8d11-fb1909ecf87d","Type":"ContainerDied","Data":"d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699"} Nov 03 20:46:39.653149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.652938 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" event={"ID":"1cb44027-53ca-4bd8-8d11-fb1909ecf87d","Type":"ContainerDied","Data":"ff03d40c995fa840e3f5ca8ca71c0d60bb1b82c0487565c04dca084bafcbf33d"} Nov 03 20:46:39.653149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.652957 5815 scope.go:122] "RemoveContainer" containerID="d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699" Nov 03 20:46:39.653149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.652897 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225" Nov 03 20:46:39.697330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.697291 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body= Nov 03 20:46:39.697550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.697349 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" Nov 03 20:46:39.704161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.704129 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225"] Nov 03 20:46:39.704673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.704650 5815 scope.go:122] "RemoveContainer" containerID="d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699" Nov 03 20:46:39.705048 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:46:39.705020 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699\": container with ID starting with d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699 not found: ID does not exist" containerID="d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699" Nov 03 20:46:39.705153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.705058 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699"} err="failed to get container status \"d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699\": rpc error: code = NotFound desc = could not find container \"d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699\": container with ID starting with d075909be39ed523e3d95f30c6e6703285016ab00847c79d878b9f8137221699 not found: ID does not exist" Nov 03 20:46:39.710023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:39.709999 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-cw225"] Nov 03 20:46:40.590922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:40.590878 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" path="/var/lib/kubelet/pods/1cb44027-53ca-4bd8-8d11-fb1909ecf87d/volumes" Nov 03 20:46:40.791546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:40.791478 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:40.793112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:40.793088 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:40.793931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:40.793908 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:46:43.860616 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:46:43.860577 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:46:45.165720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:45.165688 5815 scope.go:122] "RemoveContainer" containerID="0eab0992f8d92c24db4a5328031bd4dcbb4f7013d41d5b667a4d84db27820779" Nov 03 20:46:45.177727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:45.177712 5815 scope.go:122] "RemoveContainer" containerID="d60bdb6062c8f553dbd1deefe89e9ecf3fb9ca4b73a1cc6784695983be37bd81" Nov 03 20:46:45.188827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:45.188796 5815 scope.go:122] "RemoveContainer" containerID="76bf921cb9def7f407bd1e14adf61f814283137ab1e8cec22473f28b826a9bfc" Nov 03 20:46:45.200206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:45.200186 5815 scope.go:122] "RemoveContainer" containerID="6232267ddb795c1001abe53b05930d5f57b61a4582a4027423872278b354ec70" Nov 03 20:46:45.211346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:45.211321 5815 scope.go:122] "RemoveContainer" containerID="d97d8bf40e947b73943df8eb3d394d40747c7bfbc6de846f5a029775b11049a2" Nov 03 20:46:46.404831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:46.404793 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7"] Nov 03 20:46:46.405356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:46.404991 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" podUID="1ad21e0b-47dd-47c8-8e58-623a40229bb4" containerName="sim" containerID="cri-o://72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30" gracePeriod=30 Nov 03 20:46:46.961375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:46.961352 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:46:47.058935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.058897 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls\" (UniqueName: \"kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls\") pod \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " Nov 03 20:46:47.059136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.059075 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6\" (UniqueName: \"kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6\") pod \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\" (UID: \"1ad21e0b-47dd-47c8-8e58-623a40229bb4\") " Nov 03 20:46:47.061538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.061493 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6" pod "1ad21e0b-47dd-47c8-8e58-623a40229bb4" (UID: "1ad21e0b-47dd-47c8-8e58-623a40229bb4"). InnerVolumeSpecName "kube-api-access-sjrn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:47.061538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.061519 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls" pod "1ad21e0b-47dd-47c8-8e58-623a40229bb4" (UID: "1ad21e0b-47dd-47c8-8e58-623a40229bb4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:47.161466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.161434 5815 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1ad21e0b-47dd-47c8-8e58-623a40229bb4-tls\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:47.161642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.161470 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjrn6\" (UniqueName: \"kubernetes.io/projected/1ad21e0b-47dd-47c8-8e58-623a40229bb4-kube-api-access-sjrn6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:47.379023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.378935 5815 generic.go:356] "Generic (PLEG): container finished" podID="1ad21e0b-47dd-47c8-8e58-623a40229bb4" containerID="72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30" exitCode=0 Nov 03 20:46:47.379023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.378983 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" Nov 03 20:46:47.379023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.378994 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" event={"ID":"1ad21e0b-47dd-47c8-8e58-623a40229bb4","Type":"ContainerDied","Data":"72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30"} Nov 03 20:46:47.379023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.379026 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7" event={"ID":"1ad21e0b-47dd-47c8-8e58-623a40229bb4","Type":"ContainerDied","Data":"674e17884f3637e44221795349f16ded6ae61129fe957bcfc49d7842b698a05e"} Nov 03 20:46:47.379345 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.379042 5815 scope.go:122] "RemoveContainer" containerID="72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30" Nov 03 20:46:47.401384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.401359 5815 scope.go:122] "RemoveContainer" containerID="72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30" Nov 03 20:46:47.401779 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:46:47.401753 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30\": container with ID starting with 72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30 not found: ID does not exist" containerID="72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30" Nov 03 20:46:47.401873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.401788 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30"} err="failed to get container status \"72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30\": rpc error: code = NotFound desc = could not find container \"72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30\": container with ID starting with 72e629460ee2bb7cb1377d7ea1694ffdd332566e727ca26e782b3b6e3183db30 not found: ID does not exist" Nov 03 20:46:47.414033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.414007 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7"] Nov 03 20:46:47.421581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.421553 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-98870d/e2e-usage-sim-b9c8f1-5bdf74b499-vs6f7"] Nov 03 20:46:47.862072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:47.862039 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w"] Nov 03 20:46:48.548590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:48.548559 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5"] Nov 03 20:46:48.590811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:48.590778 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad21e0b-47dd-47c8-8e58-623a40229bb4" path="/var/lib/kubelet/pods/1ad21e0b-47dd-47c8-8e58-623a40229bb4/volumes" Nov 03 20:46:52.863178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:52.863133 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" podUID="74a4d719-b8cc-4080-910a-ede38831cbfe" containerName="payload-processing" containerID="cri-o://acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4" gracePeriod=25 Nov 03 20:46:53.253544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.253501 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:46:53.320910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.320881 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials\") pod \"74a4d719-b8cc-4080-910a-ede38831cbfe\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " Nov 03 20:46:53.321154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.321135 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume\") pod \"74a4d719-b8cc-4080-910a-ede38831cbfe\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " Nov 03 20:46:53.321274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.321257 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn\") pod \"74a4d719-b8cc-4080-910a-ede38831cbfe\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " Nov 03 20:46:53.321353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.321291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume\") pod \"74a4d719-b8cc-4080-910a-ede38831cbfe\" (UID: \"74a4d719-b8cc-4080-910a-ede38831cbfe\") " Nov 03 20:46:53.321758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.321714 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume" pod "74a4d719-b8cc-4080-910a-ede38831cbfe" (UID: "74a4d719-b8cc-4080-910a-ede38831cbfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:53.321758 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.321728 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume" pod "74a4d719-b8cc-4080-910a-ede38831cbfe" (UID: "74a4d719-b8cc-4080-910a-ede38831cbfe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:53.322698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.322675 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:53.322770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.322705 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74a4d719-b8cc-4080-910a-ede38831cbfe-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:53.323855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.323831 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn" pod "74a4d719-b8cc-4080-910a-ede38831cbfe" (UID: "74a4d719-b8cc-4080-910a-ede38831cbfe"). InnerVolumeSpecName "kube-api-access-8w9jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:53.324042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.324025 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials" pod "74a4d719-b8cc-4080-910a-ede38831cbfe" (UID: "74a4d719-b8cc-4080-910a-ede38831cbfe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:53.424997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.424963 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w9jn\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-kube-api-access-8w9jn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:53.424997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.424993 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74a4d719-b8cc-4080-910a-ede38831cbfe-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:53.550311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.550222 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" podUID="33fcacca-d0b1-4d6e-9729-0a92786838c6" containerName="payload-processing" containerID="cri-o://0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5" gracePeriod=25 Nov 03 20:46:53.653134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.653095 5815 generic.go:356] "Generic (PLEG): container finished" podID="74a4d719-b8cc-4080-910a-ede38831cbfe" containerID="acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4" exitCode=0 Nov 03 20:46:53.653843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.653559 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" event={"ID":"74a4d719-b8cc-4080-910a-ede38831cbfe","Type":"ContainerDied","Data":"acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4"} Nov 03 20:46:53.653843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.653628 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" Nov 03 20:46:53.653843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.653673 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w" event={"ID":"74a4d719-b8cc-4080-910a-ede38831cbfe","Type":"ContainerDied","Data":"b33252742dc378e66c0d22cca54ec34db5a6f4725c3edf34f865980a1205c008"} Nov 03 20:46:53.653843 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.653714 5815 scope.go:122] "RemoveContainer" containerID="acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4" Nov 03 20:46:53.668890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.668843 5815 scope.go:122] "RemoveContainer" containerID="acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4" Nov 03 20:46:53.669272 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:46:53.669249 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4\": container with ID starting with acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4 not found: ID does not exist" containerID="acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4" Nov 03 20:46:53.669362 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.669282 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4"} err="failed to get container status \"acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4\": rpc error: code = NotFound desc = could not find container \"acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4\": container with ID starting with acdf717e7eacec7555e50b82ccf22c2997d3489a065936cdd2309c13ace051d4 not found: ID does not exist" Nov 03 20:46:53.684109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.684083 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w"] Nov 03 20:46:53.692680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.692643 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-69f45m77w"] Nov 03 20:46:53.999706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:53.999675 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:46:54.153277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.153204 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5\") pod \"33fcacca-d0b1-4d6e-9729-0a92786838c6\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " Nov 03 20:46:54.153440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.153314 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume\") pod \"33fcacca-d0b1-4d6e-9729-0a92786838c6\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " Nov 03 20:46:54.153658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.153630 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials\") pod \"33fcacca-d0b1-4d6e-9729-0a92786838c6\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " Nov 03 20:46:54.153811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.153787 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume" pod "33fcacca-d0b1-4d6e-9729-0a92786838c6" (UID: "33fcacca-d0b1-4d6e-9729-0a92786838c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:54.153811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.153798 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume\") pod \"33fcacca-d0b1-4d6e-9729-0a92786838c6\" (UID: \"33fcacca-d0b1-4d6e-9729-0a92786838c6\") " Nov 03 20:46:54.154115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.154089 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume" pod "33fcacca-d0b1-4d6e-9729-0a92786838c6" (UID: "33fcacca-d0b1-4d6e-9729-0a92786838c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:54.154600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.154580 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:54.154662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.154603 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33fcacca-d0b1-4d6e-9729-0a92786838c6-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:54.155706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.155682 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5" pod "33fcacca-d0b1-4d6e-9729-0a92786838c6" (UID: "33fcacca-d0b1-4d6e-9729-0a92786838c6"). InnerVolumeSpecName "kube-api-access-czvb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:54.155859 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.155839 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials" pod "33fcacca-d0b1-4d6e-9729-0a92786838c6" (UID: "33fcacca-d0b1-4d6e-9729-0a92786838c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:54.255893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.255861 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:54.255893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.255892 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czvb5\" (UniqueName: \"kubernetes.io/projected/33fcacca-d0b1-4d6e-9729-0a92786838c6-kube-api-access-czvb5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:54.589524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.589484 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a4d719-b8cc-4080-910a-ede38831cbfe" path="/var/lib/kubelet/pods/74a4d719-b8cc-4080-910a-ede38831cbfe/volumes" Nov 03 20:46:54.665466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.665425 5815 generic.go:356] "Generic (PLEG): container finished" podID="33fcacca-d0b1-4d6e-9729-0a92786838c6" containerID="0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5" exitCode=0 Nov 03 20:46:54.665647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.665479 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" Nov 03 20:46:54.665647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.665547 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" event={"ID":"33fcacca-d0b1-4d6e-9729-0a92786838c6","Type":"ContainerDied","Data":"0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5"} Nov 03 20:46:54.665647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.665604 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5" event={"ID":"33fcacca-d0b1-4d6e-9729-0a92786838c6","Type":"ContainerDied","Data":"30f5142d0f02411edd11dc2d0715c126bed4bba276841f921b906527027643fe"} Nov 03 20:46:54.665647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.665633 5815 scope.go:122] "RemoveContainer" containerID="0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5" Nov 03 20:46:54.679682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.679665 5815 scope.go:122] "RemoveContainer" containerID="0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5" Nov 03 20:46:54.679989 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:46:54.679961 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5\": container with ID starting with 0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5 not found: ID does not exist" containerID="0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5" Nov 03 20:46:54.680148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.680008 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5"} err="failed to get container status \"0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5\": rpc error: code = NotFound desc = could not find container \"0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5\": container with ID starting with 0886166b56489d9f481d757cb046ae91ec0646a0329f24c334dc269ce76ca6e5 not found: ID does not exist" Nov 03 20:46:54.700019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.699989 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5"] Nov 03 20:46:54.705429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:54.705405 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-5c98jzwr5"] Nov 03 20:46:55.587726 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.587691 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z"] Nov 03 20:46:55.588969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.588942 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1ad21e0b-47dd-47c8-8e58-623a40229bb4" containerName="sim" Nov 03 20:46:55.588969 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.588969 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad21e0b-47dd-47c8-8e58-623a40229bb4" containerName="sim" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.588996 5815 cpu_manager.go:414] "containerMap: removing container" podUID="33fcacca-d0b1-4d6e-9729-0a92786838c6" containerName="payload-processing" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589003 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33fcacca-d0b1-4d6e-9729-0a92786838c6" containerName="payload-processing" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589033 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589041 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589091 5815 cpu_manager.go:414] "containerMap: removing container" podUID="74a4d719-b8cc-4080-910a-ede38831cbfe" containerName="payload-processing" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589098 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a4d719-b8cc-4080-910a-ede38831cbfe" containerName="payload-processing" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589115 5815 cpu_manager.go:414] "containerMap: removing container" podUID="abd5092d-8651-4d95-a794-30c12767a10d" containerName="revoke-keys" Nov 03 20:46:55.589160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589122 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd5092d-8651-4d95-a794-30c12767a10d" containerName="revoke-keys" Nov 03 20:46:55.589660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589324 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a4d719-b8cc-4080-910a-ede38831cbfe" containerName="payload-processing" Nov 03 20:46:55.589660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589337 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd5092d-8651-4d95-a794-30c12767a10d" containerName="revoke-keys" Nov 03 20:46:55.589660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589357 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad21e0b-47dd-47c8-8e58-623a40229bb4" containerName="sim" Nov 03 20:46:55.589660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589368 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="33fcacca-d0b1-4d6e-9729-0a92786838c6" containerName="payload-processing" Nov 03 20:46:55.589660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.589381 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb44027-53ca-4bd8-8d11-fb1909ecf87d" containerName="istio-proxy" Nov 03 20:46:55.597723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.594489 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.609712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.604280 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bf7965-openshift-default-dockercfg-dxxdr\"" Nov 03 20:46:55.633484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.633437 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z"] Nov 03 20:46:55.698809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.698778 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.698826 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699004 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699023 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699072 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699102 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699119 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699241 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.699371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.699297 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tks8\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.808092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.808052 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.808410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.808376 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.808733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.808700 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.808892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.808875 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.809084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.809066 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.809232 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.809217 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.822057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.822034 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.822397 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.822379 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.822715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.822693 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tks8\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.823879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.823853 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.824256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.824237 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.824411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.821255 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.825187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.825165 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.825947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.825925 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.882579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.874304 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.883333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.883276 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.891613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.891565 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tks8\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.916531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.913053 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo\") pod \"e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:55.943030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:55.942765 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:56.555849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:56.553130 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z"] Nov 03 20:46:56.636892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:56.636838 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:46:56.637296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:56.637007 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:46:56.637296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:56.637092 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:46:56.649574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:56.649434 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33fcacca-d0b1-4d6e-9729-0a92786838c6" path="/var/lib/kubelet/pods/33fcacca-d0b1-4d6e-9729-0a92786838c6/volumes" Nov 03 20:46:56.710532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:56.710475 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" event={"ID":"6a47b882-1e6c-4aef-8b71-9e1b8629b857","Type":"ContainerStarted","Data":"65f4ef6347e7e33c6543acf06b8d9e91b7326effc0086f1c5ba8fb04bb508c0c"} Nov 03 20:46:57.722828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:57.722790 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" event={"ID":"6a47b882-1e6c-4aef-8b71-9e1b8629b857","Type":"ContainerStarted","Data":"b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9"} Nov 03 20:46:57.750174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:57.750117 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podStartSLOduration=2.750104178 podStartE2EDuration="2.750104178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:57.747756217 +0000 UTC m=+2727.278101893" watchObservedRunningTime="2026-11-03 20:46:57.750104178 +0000 UTC m=+2727.280449849" Nov 03 20:46:57.947346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:57.945262 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:57.970487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:57.970441 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:58.733284 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:58.733250 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:46:58.734272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:46:58.734237 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:47:07.138532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:07.137586 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" probeResult="failure" output="Get \"https://10.132.1.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:07.241150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:07.238229 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 20:47:07.322537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:07.321898 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4"] Nov 03 20:47:07.322753 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:07.322584 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" containerID="cri-o://fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192" gracePeriod=30 Nov 03 20:47:09.529200 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.529162 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h"] Nov 03 20:47:09.537601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.537576 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.541364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.541334 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h"] Nov 03 20:47:09.542416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.542395 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:47:09.542416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.542391 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:09.590375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.590351 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.590598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.590409 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9r5\" (UniqueName: \"kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.590715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.590695 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.692430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.692393 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.692657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.692473 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9r5\" (UniqueName: \"kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.693306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.692772 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.693791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.693763 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.695291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.695267 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.704743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.704722 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9r5\" (UniqueName: \"kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5\") pod \"maas-api-revoke-sub-221e46b0-cww4h\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:09.862385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:09.862297 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:10.151660 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:47:10.151627 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd9eef0_c0e5_414d_a227_26b7a0e12c2a.slice/crio-baa3665c3640eef7f256f7004ff66b420bec2b0cfece812882bc712f2c6bac19 WatchSource:0}: Error finding container baa3665c3640eef7f256f7004ff66b420bec2b0cfece812882bc712f2c6bac19: Status 404 returned error can't find the container with id baa3665c3640eef7f256f7004ff66b420bec2b0cfece812882bc712f2c6bac19 Nov 03 20:47:10.152437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:10.152412 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h"] Nov 03 20:47:11.147540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:11.146993 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" event={"ID":"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a","Type":"ContainerStarted","Data":"733a95c14860d1655ee76fc003a1080a4bab02b251216bf21a67c6786ec9422a"} Nov 03 20:47:11.147540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:11.147070 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" event={"ID":"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a","Type":"ContainerStarted","Data":"baa3665c3640eef7f256f7004ff66b420bec2b0cfece812882bc712f2c6bac19"} Nov 03 20:47:11.212054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:11.211975 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" podStartSLOduration=2.21195489 podStartE2EDuration="2.21195489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:11.182896462 +0000 UTC m=+2740.713242135" watchObservedRunningTime="2026-11-03 20:47:11.21195489 +0000 UTC m=+2740.742300578" Nov 03 20:47:11.759006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:11.758970 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:47:11.759211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:11.759029 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:47:12.158501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.158412 5815 generic.go:356] "Generic (PLEG): container finished" podID="7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" containerID="733a95c14860d1655ee76fc003a1080a4bab02b251216bf21a67c6786ec9422a" exitCode=0 Nov 03 20:47:12.158930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.158489 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" event={"ID":"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a","Type":"ContainerDied","Data":"733a95c14860d1655ee76fc003a1080a4bab02b251216bf21a67c6786ec9422a"} Nov 03 20:47:12.731981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.731955 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:47:12.764588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.764557 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.764767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.764631 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.764767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.764684 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.764767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.764742 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.765427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.764834 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.765427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.764905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.765427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.765049 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.765427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.765098 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.765427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.765253 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert\") pod \"1e23e11c-6c2f-4130-b713-d8615d6615b7\" (UID: \"1e23e11c-6c2f-4130-b713-d8615d6615b7\") " Nov 03 20:47:12.765771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.765543 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:12.765826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.765448 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:12.765949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.765926 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:12.766172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.766154 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:12.766973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.766892 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.766973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.766921 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.766973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.766936 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e23e11c-6c2f-4130-b713-d8615d6615b7-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.766973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.766951 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.767487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.767463 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:12.769895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.769857 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:47:12.771676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.771639 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:12.772226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.772192 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:12.773226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.773202 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb" pod "1e23e11c-6c2f-4130-b713-d8615d6615b7" (UID: "1e23e11c-6c2f-4130-b713-d8615d6615b7"). InnerVolumeSpecName "kube-api-access-4r7lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:12.868696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.868665 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7lb\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-kube-api-access-4r7lb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.868696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.868696 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.868893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.868713 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e23e11c-6c2f-4130-b713-d8615d6615b7-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.868893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.868729 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:12.868893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:12.868743 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e23e11c-6c2f-4130-b713-d8615d6615b7-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:13.171591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.171559 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" Nov 03 20:47:13.171591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.171583 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" event={"ID":"1e23e11c-6c2f-4130-b713-d8615d6615b7","Type":"ContainerDied","Data":"fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192"} Nov 03 20:47:13.172172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.171550 5815 generic.go:356] "Generic (PLEG): container finished" podID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerID="fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192" exitCode=0 Nov 03 20:47:13.172172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.171645 5815 scope.go:122] "RemoveContainer" containerID="fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192" Nov 03 20:47:13.172172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.171775 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4" event={"ID":"1e23e11c-6c2f-4130-b713-d8615d6615b7","Type":"ContainerDied","Data":"66c93ea1b96e267a17f64e0290a6c400a9876c4683f7179a72947f3b9fe88289"} Nov 03 20:47:13.196793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.196775 5815 scope.go:122] "RemoveContainer" containerID="fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192" Nov 03 20:47:13.197183 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:47:13.197160 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192\": container with ID starting with fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192 not found: ID does not exist" containerID="fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192" Nov 03 20:47:13.197285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.197195 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192"} err="failed to get container status \"fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192\": rpc error: code = NotFound desc = could not find container \"fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192\": container with ID starting with fd37078d8fb5ef14119edce1e75bf342b2f479a0bd8d0ee6231baa7cc182b192 not found: ID does not exist" Nov 03 20:47:13.216295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.216265 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4"] Nov 03 20:47:13.223561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.223535 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qd4f4"] Nov 03 20:47:13.348311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.348278 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:13.376273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.376247 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca\") pod \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " Nov 03 20:47:13.376438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.376416 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca\") pod \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " Nov 03 20:47:13.376633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.376612 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5\" (UniqueName: \"kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5\") pod \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\" (UID: \"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a\") " Nov 03 20:47:13.377220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.377193 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca" pod "7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" (UID: "7fd9eef0-c0e5-414d-a227-26b7a0e12c2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:13.378953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.378924 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:13.384069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.384031 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5" pod "7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" (UID: "7fd9eef0-c0e5-414d-a227-26b7a0e12c2a"). InnerVolumeSpecName "kube-api-access-rn9r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:13.384446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.384422 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca" pod "7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" (UID: "7fd9eef0-c0e5-414d-a227-26b7a0e12c2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:13.486541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.484283 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn9r5\" (UniqueName: \"kubernetes.io/projected/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-kube-api-access-rn9r5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:13.486541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:13.484321 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:13.853238 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:47:13.853148 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:47:14.212067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:14.185835 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" event={"ID":"7fd9eef0-c0e5-414d-a227-26b7a0e12c2a","Type":"ContainerDied","Data":"baa3665c3640eef7f256f7004ff66b420bec2b0cfece812882bc712f2c6bac19"} Nov 03 20:47:14.212067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:14.185902 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa3665c3640eef7f256f7004ff66b420bec2b0cfece812882bc712f2c6bac19" Nov 03 20:47:14.212067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:14.185973 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h" Nov 03 20:47:14.589333 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:14.589261 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" path="/var/lib/kubelet/pods/1e23e11c-6c2f-4130-b713-d8615d6615b7/volumes" Nov 03 20:47:20.016680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.016642 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr"] Nov 03 20:47:20.017472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.017446 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" Nov 03 20:47:20.017569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.017473 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" Nov 03 20:47:20.017569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.017533 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" containerName="revoke-keys" Nov 03 20:47:20.017569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.017540 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" containerName="revoke-keys" Nov 03 20:47:20.017757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.017745 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" containerName="revoke-keys" Nov 03 20:47:20.017801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.017769 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e23e11c-6c2f-4130-b713-d8615d6615b7" containerName="istio-proxy" Nov 03 20:47:20.022202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.022184 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.027207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.027167 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:20.027207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.027184 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:47:20.029010 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.028990 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr"] Nov 03 20:47:20.091965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.091930 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.092216 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.092181 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.092438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.092414 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzbv\" (UniqueName: \"kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.195172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.195140 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.195361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.195308 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.195361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.195346 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzbv\" (UniqueName: \"kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.196158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.196127 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.198046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.198015 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.209343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.209319 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzbv\" (UniqueName: \"kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv\") pod \"maas-api-revoke-sub-ff493c9c-kl8fr\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.340132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.340045 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:20.724551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:20.724475 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr"] Nov 03 20:47:21.403584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:21.401746 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" event={"ID":"eb233ea4-b8fa-4c8e-b651-436450856cf3","Type":"ContainerStarted","Data":"d6f255425b3194d126b3e1ed6a95329b80bba23938725dcb272ffef91eaac09b"} Nov 03 20:47:21.403584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:21.401822 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" event={"ID":"eb233ea4-b8fa-4c8e-b651-436450856cf3","Type":"ContainerStarted","Data":"491e2fdd375777d8933528d2483cd211fafa709adfacf5b2032147a6f8d06935"} Nov 03 20:47:21.435034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:21.434953 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" podStartSLOduration=2.434936242 podStartE2EDuration="2.434936242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:21.430111415 +0000 UTC m=+2750.960457080" watchObservedRunningTime="2026-11-03 20:47:21.434936242 +0000 UTC m=+2750.965281917" Nov 03 20:47:22.475423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:22.475376 5815 generic.go:356] "Generic (PLEG): container finished" podID="eb233ea4-b8fa-4c8e-b651-436450856cf3" containerID="d6f255425b3194d126b3e1ed6a95329b80bba23938725dcb272ffef91eaac09b" exitCode=0 Nov 03 20:47:22.476004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:22.475674 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" event={"ID":"eb233ea4-b8fa-4c8e-b651-436450856cf3","Type":"ContainerDied","Data":"d6f255425b3194d126b3e1ed6a95329b80bba23938725dcb272ffef91eaac09b"} Nov 03 20:47:23.677624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.677599 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:23.802813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.802729 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca\") pod \"eb233ea4-b8fa-4c8e-b651-436450856cf3\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " Nov 03 20:47:23.802988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.802958 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv\" (UniqueName: \"kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv\") pod \"eb233ea4-b8fa-4c8e-b651-436450856cf3\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " Nov 03 20:47:23.803158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.803131 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca" pod "eb233ea4-b8fa-4c8e-b651-436450856cf3" (UID: "eb233ea4-b8fa-4c8e-b651-436450856cf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:23.803229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.803182 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca\") pod \"eb233ea4-b8fa-4c8e-b651-436450856cf3\" (UID: \"eb233ea4-b8fa-4c8e-b651-436450856cf3\") " Nov 03 20:47:23.804089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.804058 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:23.805485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.805457 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv" pod "eb233ea4-b8fa-4c8e-b651-436450856cf3" (UID: "eb233ea4-b8fa-4c8e-b651-436450856cf3"). InnerVolumeSpecName "kube-api-access-7lzbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:23.805693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.805670 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca" pod "eb233ea4-b8fa-4c8e-b651-436450856cf3" (UID: "eb233ea4-b8fa-4c8e-b651-436450856cf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:23.906256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.906220 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzbv\" (UniqueName: \"kubernetes.io/projected/eb233ea4-b8fa-4c8e-b651-436450856cf3-kube-api-access-7lzbv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:23.906419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:23.906259 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb233ea4-b8fa-4c8e-b651-436450856cf3-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:24.498853 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:24.498826 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" Nov 03 20:47:24.499046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:24.498874 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr" event={"ID":"eb233ea4-b8fa-4c8e-b651-436450856cf3","Type":"ContainerDied","Data":"491e2fdd375777d8933528d2483cd211fafa709adfacf5b2032147a6f8d06935"} Nov 03 20:47:24.499046 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:24.498929 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491e2fdd375777d8933528d2483cd211fafa709adfacf5b2032147a6f8d06935" Nov 03 20:47:27.726635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:27.726601 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h"] Nov 03 20:47:27.727088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:27.726853 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" containerID="cri-o://0e33d1677f3e18f1724062f459bdc18edcc9d6fd5a13b7f3cfd7bc3615b4be57" gracePeriod=30 Nov 03 20:47:32.035896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.035861 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf"] Nov 03 20:47:32.088406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.062922 5815 cpu_manager.go:414] "containerMap: removing container" podUID="eb233ea4-b8fa-4c8e-b651-436450856cf3" containerName="revoke-keys" Nov 03 20:47:32.088704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.088678 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb233ea4-b8fa-4c8e-b651-436450856cf3" containerName="revoke-keys" Nov 03 20:47:32.104534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.089202 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb233ea4-b8fa-4c8e-b651-436450856cf3" containerName="revoke-keys" Nov 03 20:47:32.193551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.187250 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf"] Nov 03 20:47:32.201134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.197813 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.201134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.200630 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.201134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.200888 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.201134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.200921 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.201134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.200993 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkmh\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.252419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.252374 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk"] Nov 03 20:47:32.263339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.263316 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.265146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.265117 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk"] Nov 03 20:47:32.270959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.270489 5815 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:47:32.270959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.270556 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:47:32.303139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.303058 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.303139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.303112 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.303369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.303170 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkmh\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.303425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.303382 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.304209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.304182 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.304961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.304938 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.306094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.306067 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.318821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.318792 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkmh\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.347980 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.347953 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:32.348108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.348015 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:47:32.404767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.404740 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.404994 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.404975 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2b5z\" (UniqueName: \"kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.405041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.405025 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.509067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.509030 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.509791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.509527 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b5z\" (UniqueName: \"kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.509791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.509618 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.515524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.510923 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.524240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.524218 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.524781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.524749 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b5z\" (UniqueName: \"kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z\") pod \"maas-api-revoke-sub-c10ceea9-v9pkk\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.592774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.580068 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:32.602984 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.602901 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:32.991740 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:47:32.991707 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdba816_d054_4202_befa_90955295a454.slice/crio-4d6530a0dd47e951722eccbb9aedc4b3b112de02885b3d8da371e66952e9aba9 WatchSource:0}: Error finding container 4d6530a0dd47e951722eccbb9aedc4b3b112de02885b3d8da371e66952e9aba9: Status 404 returned error can't find the container with id 4d6530a0dd47e951722eccbb9aedc4b3b112de02885b3d8da371e66952e9aba9 Nov 03 20:47:32.992651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.992610 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf"] Nov 03 20:47:32.994371 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:47:32.994341 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b60efa_e35e_4659_b74b_5ebb0b67a6f2.slice/crio-60e5534c0d20eaef7645202a1f2e0576dabfa1f6d17cddcc5991839aaaf47b28 WatchSource:0}: Error finding container 60e5534c0d20eaef7645202a1f2e0576dabfa1f6d17cddcc5991839aaaf47b28: Status 404 returned error can't find the container with id 60e5534c0d20eaef7645202a1f2e0576dabfa1f6d17cddcc5991839aaaf47b28 Nov 03 20:47:32.999535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:32.999315 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk"] Nov 03 20:47:33.223594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.223563 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" event={"ID":"8fdba816-d054-4202-befa-90955295a454","Type":"ContainerStarted","Data":"0a102317872cb013dbd08480479ffe6ce90e713ff34361daa8acaea3be676c09"} Nov 03 20:47:33.224002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.223618 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" event={"ID":"8fdba816-d054-4202-befa-90955295a454","Type":"ContainerStarted","Data":"4d6530a0dd47e951722eccbb9aedc4b3b112de02885b3d8da371e66952e9aba9"} Nov 03 20:47:33.224002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.223677 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:33.225620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.225594 5815 generic.go:356] "Generic (PLEG): container finished" podID="35db10e6-9717-4145-be19-db5022ebfe24" containerID="0e33d1677f3e18f1724062f459bdc18edcc9d6fd5a13b7f3cfd7bc3615b4be57" exitCode=0 Nov 03 20:47:33.225761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.225665 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" event={"ID":"35db10e6-9717-4145-be19-db5022ebfe24","Type":"ContainerDied","Data":"0e33d1677f3e18f1724062f459bdc18edcc9d6fd5a13b7f3cfd7bc3615b4be57"} Nov 03 20:47:33.227420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.227356 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" event={"ID":"98b60efa-e35e-4659-b74b-5ebb0b67a6f2","Type":"ContainerStarted","Data":"60e5534c0d20eaef7645202a1f2e0576dabfa1f6d17cddcc5991839aaaf47b28"} Nov 03 20:47:33.251377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.251327 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" podStartSLOduration=2.25130918 podStartE2EDuration="2.25130918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:33.242499696 +0000 UTC m=+2762.772845369" watchObservedRunningTime="2026-11-03 20:47:33.25130918 +0000 UTC m=+2762.781654852" Nov 03 20:47:33.520696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.520636 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:47:33.559361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559328 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559369 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559490 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559570 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559703 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559695 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559762 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559824 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.559915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559857 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:33.559976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.559957 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.560099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.560079 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:33.560155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.560094 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:33.560302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.560210 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo\") pod \"35db10e6-9717-4145-be19-db5022ebfe24\" (UID: \"35db10e6-9717-4145-be19-db5022ebfe24\") " Nov 03 20:47:33.560640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.560275 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:33.560640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.560284 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:33.561223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.561198 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.561327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.561230 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.561327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.561243 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.561327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.561256 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35db10e6-9717-4145-be19-db5022ebfe24-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.561327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.561271 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.568223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.568198 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "kube-api-access-gzppg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:33.569136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.569111 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:33.569647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.569618 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:47:33.569788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.569767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token" pod "35db10e6-9717-4145-be19-db5022ebfe24" (UID: "35db10e6-9717-4145-be19-db5022ebfe24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:33.667130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.664574 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35db10e6-9717-4145-be19-db5022ebfe24-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.667130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.664612 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzppg\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-kube-api-access-gzppg\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.667130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.664629 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db10e6-9717-4145-be19-db5022ebfe24-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:33.667130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:33.664643 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35db10e6-9717-4145-be19-db5022ebfe24-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:34.238761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.238733 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" Nov 03 20:47:34.238761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.238733 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h" event={"ID":"35db10e6-9717-4145-be19-db5022ebfe24","Type":"ContainerDied","Data":"0d3ba13ab745fa15f8b026805c7530874a0af022e1582290aa8f509f60914ae4"} Nov 03 20:47:34.239261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.238819 5815 scope.go:122] "RemoveContainer" containerID="0e33d1677f3e18f1724062f459bdc18edcc9d6fd5a13b7f3cfd7bc3615b4be57" Nov 03 20:47:34.241033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.241004 5815 generic.go:356] "Generic (PLEG): container finished" podID="98b60efa-e35e-4659-b74b-5ebb0b67a6f2" containerID="55702f0864ad41237b8c7283194db224d7d3eda7a9f22f2529e6803c8c5110cd" exitCode=0 Nov 03 20:47:34.241151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.241041 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" event={"ID":"98b60efa-e35e-4659-b74b-5ebb0b67a6f2","Type":"ContainerDied","Data":"55702f0864ad41237b8c7283194db224d7d3eda7a9f22f2529e6803c8c5110cd"} Nov 03 20:47:34.280332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.280308 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h"] Nov 03 20:47:34.284927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.284907 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fx92h"] Nov 03 20:47:34.592176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:34.592141 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35db10e6-9717-4145-be19-db5022ebfe24" path="/var/lib/kubelet/pods/35db10e6-9717-4145-be19-db5022ebfe24/volumes" Nov 03 20:47:35.547402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.547380 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:35.604840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.604802 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca\") pod \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " Nov 03 20:47:35.605062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.604984 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z\" (UniqueName: \"kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z\") pod \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " Nov 03 20:47:35.605062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.605017 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca\") pod \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\" (UID: \"98b60efa-e35e-4659-b74b-5ebb0b67a6f2\") " Nov 03 20:47:35.605716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.605687 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca" pod "98b60efa-e35e-4659-b74b-5ebb0b67a6f2" (UID: "98b60efa-e35e-4659-b74b-5ebb0b67a6f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:35.606106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.606083 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:35.609552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.608895 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z" pod "98b60efa-e35e-4659-b74b-5ebb0b67a6f2" (UID: "98b60efa-e35e-4659-b74b-5ebb0b67a6f2"). InnerVolumeSpecName "kube-api-access-r2b5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:35.610066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.610043 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca" pod "98b60efa-e35e-4659-b74b-5ebb0b67a6f2" (UID: "98b60efa-e35e-4659-b74b-5ebb0b67a6f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:35.707793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.707761 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2b5z\" (UniqueName: \"kubernetes.io/projected/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-kube-api-access-r2b5z\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:35.707793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:35.707791 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98b60efa-e35e-4659-b74b-5ebb0b67a6f2-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:36.319568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:36.310635 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" event={"ID":"98b60efa-e35e-4659-b74b-5ebb0b67a6f2","Type":"ContainerDied","Data":"60e5534c0d20eaef7645202a1f2e0576dabfa1f6d17cddcc5991839aaaf47b28"} Nov 03 20:47:36.319568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:36.310731 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk" Nov 03 20:47:36.319568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:36.310747 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e5534c0d20eaef7645202a1f2e0576dabfa1f6d17cddcc5991839aaaf47b28" Nov 03 20:47:42.723618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.723581 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh"] Nov 03 20:47:42.724422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.724406 5815 cpu_manager.go:414] "containerMap: removing container" podUID="98b60efa-e35e-4659-b74b-5ebb0b67a6f2" containerName="revoke-keys" Nov 03 20:47:42.724469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.724423 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98b60efa-e35e-4659-b74b-5ebb0b67a6f2" containerName="revoke-keys" Nov 03 20:47:42.724469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.724441 5815 cpu_manager.go:414] "containerMap: removing container" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" Nov 03 20:47:42.724469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.724448 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" Nov 03 20:47:42.724635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.724623 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="98b60efa-e35e-4659-b74b-5ebb0b67a6f2" containerName="revoke-keys" Nov 03 20:47:42.724677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.724639 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="35db10e6-9717-4145-be19-db5022ebfe24" containerName="istio-proxy" Nov 03 20:47:42.728754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.728735 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.733425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.733393 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:42.734704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.734686 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:47:42.739155 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.739132 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh"] Nov 03 20:47:42.810210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.810173 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.810465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.810446 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.810690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.810674 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncmw\" (UniqueName: \"kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.911888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.911858 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.912061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.911999 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncmw\" (UniqueName: \"kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.912120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.912097 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.912701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.912674 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.914783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.914761 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:42.924468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:42.924447 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncmw\" (UniqueName: \"kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw\") pod \"maas-api-revoke-sub-832a34d5-hp8mh\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:43.048179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:43.048118 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:43.854432 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:47:43.854391 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:47:43.860152 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:47:43.860125 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03aec59_a797_46b1_a32b_48df42809845.slice/crio-b3e3346444c8814079a9fd33f94e8f03d5b4db86cb2856894df445a1e0e53041 WatchSource:0}: Error finding container b3e3346444c8814079a9fd33f94e8f03d5b4db86cb2856894df445a1e0e53041: Status 404 returned error can't find the container with id b3e3346444c8814079a9fd33f94e8f03d5b4db86cb2856894df445a1e0e53041 Nov 03 20:47:43.861426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:43.861404 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh"] Nov 03 20:47:44.252567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:44.245385 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:47:44.321186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:44.321148 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64"] Nov 03 20:47:44.647385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:44.647292 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" event={"ID":"c03aec59-a797-46b1-a32b-48df42809845","Type":"ContainerStarted","Data":"6c67059eeb0dd78be729ce9230b9dfaa58dce1c55f32d5bdaf91c552096b27a9"} Nov 03 20:47:44.647385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:44.647352 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" event={"ID":"c03aec59-a797-46b1-a32b-48df42809845","Type":"ContainerStarted","Data":"b3e3346444c8814079a9fd33f94e8f03d5b4db86cb2856894df445a1e0e53041"} Nov 03 20:47:44.667350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:44.667290 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" podStartSLOduration=2.667271916 podStartE2EDuration="2.667271916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:44.665321752 +0000 UTC m=+2774.195667427" watchObservedRunningTime="2026-11-03 20:47:44.667271916 +0000 UTC m=+2774.197617594" Nov 03 20:47:44.912767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:44.912651 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw"] Nov 03 20:47:45.309438 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.309405 5815 scope.go:122] "RemoveContainer" containerID="df9132ec58b788a5692010013d6ee9cc136094cf19634f977e59667b9021c45f" Nov 03 20:47:45.322108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.322090 5815 scope.go:122] "RemoveContainer" containerID="f2bb6fc98396467f174e21392960d743fcafd3962aa02316102efdfe1833e941" Nov 03 20:47:45.333912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.333887 5815 scope.go:122] "RemoveContainer" containerID="a8a52edf57c8cc022d29b0cbeba78ace1b012b026ccbc8d0619d157656f69bba" Nov 03 20:47:45.344591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.344574 5815 scope.go:122] "RemoveContainer" containerID="8d464f71112fa5a0837c28475f5f63505a0a707017e82b4e8ab594692c37a3c0" Nov 03 20:47:45.355121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.355092 5815 scope.go:122] "RemoveContainer" containerID="630685e305f2051237082e07178df400d3c584c8715a4cc63364a060faafae6f" Nov 03 20:47:45.366649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.366628 5815 scope.go:122] "RemoveContainer" containerID="48c827816c3b8853d3a1d55b0fc99bd26e559c6954aff9439dbb1942efdd188e" Nov 03 20:47:45.381883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.381852 5815 scope.go:122] "RemoveContainer" containerID="107a14794a1b1fe080cb415ac8e63f0f5a2f466fc36ad87338e02759e00d1a2b" Nov 03 20:47:45.392707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.392675 5815 scope.go:122] "RemoveContainer" containerID="e011d5d3f57443d349bd5a37fd5b3bc3d240c7c466a6b45c58922f04f2f13142" Nov 03 20:47:45.658970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.658887 5815 generic.go:356] "Generic (PLEG): container finished" podID="c03aec59-a797-46b1-a32b-48df42809845" containerID="6c67059eeb0dd78be729ce9230b9dfaa58dce1c55f32d5bdaf91c552096b27a9" exitCode=0 Nov 03 20:47:45.658970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:45.658938 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" event={"ID":"c03aec59-a797-46b1-a32b-48df42809845","Type":"ContainerDied","Data":"6c67059eeb0dd78be729ce9230b9dfaa58dce1c55f32d5bdaf91c552096b27a9"} Nov 03 20:47:46.804542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.802759 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m"] Nov 03 20:47:46.833468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.833425 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919447 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtdx\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919553 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919599 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919644 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919705 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919865 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.919889 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.920035 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.926598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.920074 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:46.962543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:46.947856 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d7c7fd-openshift-default-dockercfg-r2g2h\"" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.024821 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025027 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025055 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025201 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025245 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025337 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtdx\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025408 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025463 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.025530 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.028926 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.029178 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.046710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.029422 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.107533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.104782 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m"] Nov 03 20:47:47.107533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.105177 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.146638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.101063 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.352562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.349266 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.353226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.353076 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.353930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.353905 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.383407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.383316 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtdx\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx\") pod \"e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.658722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.658372 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:47.732132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.731030 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" event={"ID":"c03aec59-a797-46b1-a32b-48df42809845","Type":"ContainerDied","Data":"b3e3346444c8814079a9fd33f94e8f03d5b4db86cb2856894df445a1e0e53041"} Nov 03 20:47:47.732132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.731125 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e3346444c8814079a9fd33f94e8f03d5b4db86cb2856894df445a1e0e53041" Nov 03 20:47:47.732132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.731645 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:47.848957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.848924 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca\") pod \"c03aec59-a797-46b1-a32b-48df42809845\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " Nov 03 20:47:47.849347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.848995 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca\") pod \"c03aec59-a797-46b1-a32b-48df42809845\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " Nov 03 20:47:47.849347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.849184 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw\" (UniqueName: \"kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw\") pod \"c03aec59-a797-46b1-a32b-48df42809845\" (UID: \"c03aec59-a797-46b1-a32b-48df42809845\") " Nov 03 20:47:47.849799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.849778 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca" pod "c03aec59-a797-46b1-a32b-48df42809845" (UID: "c03aec59-a797-46b1-a32b-48df42809845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:47.850085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.850059 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03aec59-a797-46b1-a32b-48df42809845-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:47.852029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.852004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca" pod "c03aec59-a797-46b1-a32b-48df42809845" (UID: "c03aec59-a797-46b1-a32b-48df42809845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:47.852443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.852426 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw" pod "c03aec59-a797-46b1-a32b-48df42809845" (UID: "c03aec59-a797-46b1-a32b-48df42809845"). InnerVolumeSpecName "kube-api-access-5ncmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:47.952725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.952655 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03aec59-a797-46b1-a32b-48df42809845-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:47.952725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.952680 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ncmw\" (UniqueName: \"kubernetes.io/projected/c03aec59-a797-46b1-a32b-48df42809845-kube-api-access-5ncmw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:48.002530 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:47:47.997806 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904186e6_a013_4ce4_ab77_0d5893e2df5e.slice/crio-fb5a6cf0e24c2f2e6c4ed9869cd5dfbb2351a23b99ce5fad3a3c95185c4e51c4 WatchSource:0}: Error finding container fb5a6cf0e24c2f2e6c4ed9869cd5dfbb2351a23b99ce5fad3a3c95185c4e51c4: Status 404 returned error can't find the container with id fb5a6cf0e24c2f2e6c4ed9869cd5dfbb2351a23b99ce5fad3a3c95185c4e51c4 Nov 03 20:47:48.002530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:47.999769 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m"] Nov 03 20:47:48.002530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.002178 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:47:48.002530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.002263 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:47:48.002530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.002310 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:47:48.745568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.745484 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" event={"ID":"904186e6-a013-4ce4-ab77-0d5893e2df5e","Type":"ContainerStarted","Data":"afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f"} Nov 03 20:47:48.745774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.745594 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" event={"ID":"904186e6-a013-4ce4-ab77-0d5893e2df5e","Type":"ContainerStarted","Data":"fb5a6cf0e24c2f2e6c4ed9869cd5dfbb2351a23b99ce5fad3a3c95185c4e51c4"} Nov 03 20:47:48.745774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.745549 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh" Nov 03 20:47:48.774846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:48.774796 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podStartSLOduration=2.7747802139999997 podStartE2EDuration="2.774780214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:48.768979513 +0000 UTC m=+2778.299325187" watchObservedRunningTime="2026-11-03 20:47:48.774780214 +0000 UTC m=+2778.305125885" Nov 03 20:47:49.322541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.322452 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" podUID="50b4bc6d-436c-4573-a82a-548f0fc55e48" containerName="payload-processing" containerID="cri-o://5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe" gracePeriod=25 Nov 03 20:47:49.627428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.627404 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:47:49.672543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.671120 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:49.719700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.719671 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials\") pod \"50b4bc6d-436c-4573-a82a-548f0fc55e48\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " Nov 03 20:47:49.720201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.720179 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume\") pod \"50b4bc6d-436c-4573-a82a-548f0fc55e48\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " Nov 03 20:47:49.720486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.720465 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume\") pod \"50b4bc6d-436c-4573-a82a-548f0fc55e48\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " Nov 03 20:47:49.720614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.720579 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9\") pod \"50b4bc6d-436c-4573-a82a-548f0fc55e48\" (UID: \"50b4bc6d-436c-4573-a82a-548f0fc55e48\") " Nov 03 20:47:49.721866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.721839 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume" pod "50b4bc6d-436c-4573-a82a-548f0fc55e48" (UID: "50b4bc6d-436c-4573-a82a-548f0fc55e48"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:49.722215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.722195 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume" pod "50b4bc6d-436c-4573-a82a-548f0fc55e48" (UID: "50b4bc6d-436c-4573-a82a-548f0fc55e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:49.726764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.726740 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9" pod "50b4bc6d-436c-4573-a82a-548f0fc55e48" (UID: "50b4bc6d-436c-4573-a82a-548f0fc55e48"). InnerVolumeSpecName "kube-api-access-vpjp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:49.739944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.739922 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:49.740699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.740675 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials" pod "50b4bc6d-436c-4573-a82a-548f0fc55e48" (UID: "50b4bc6d-436c-4573-a82a-548f0fc55e48"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:49.759847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.759820 5815 generic.go:356] "Generic (PLEG): container finished" podID="50b4bc6d-436c-4573-a82a-548f0fc55e48" containerID="5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe" exitCode=0 Nov 03 20:47:49.759977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.759885 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" Nov 03 20:47:49.759977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.759903 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" event={"ID":"50b4bc6d-436c-4573-a82a-548f0fc55e48","Type":"ContainerDied","Data":"5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe"} Nov 03 20:47:49.759977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.759958 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64" event={"ID":"50b4bc6d-436c-4573-a82a-548f0fc55e48","Type":"ContainerDied","Data":"29f2cac0a8d189622ebe9691f575e2bc231bd45b05b3ff802778dc6952ace1c4"} Nov 03 20:47:49.760137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.759986 5815 scope.go:122] "RemoveContainer" containerID="5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe" Nov 03 20:47:49.760609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.760588 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:49.761802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.761784 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:47:49.775098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.775076 5815 scope.go:122] "RemoveContainer" containerID="5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe" Nov 03 20:47:49.776401 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:47:49.776378 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe\": container with ID starting with 5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe not found: ID does not exist" containerID="5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe" Nov 03 20:47:49.776496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.776414 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe"} err="failed to get container status \"5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe\": rpc error: code = NotFound desc = could not find container \"5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe\": container with ID starting with 5554ac03e85e491673b27d7168bf6cf43c626ea4cb7f46e20f3c7434d1b28bfe not found: ID does not exist" Nov 03 20:47:49.822999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.822932 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.822999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.822958 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50b4bc6d-436c-4573-a82a-548f0fc55e48-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.822999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.822969 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpjp9\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-kube-api-access-vpjp9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.822999 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.822978 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50b4bc6d-436c-4573-a82a-548f0fc55e48-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.837632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.837040 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64"] Nov 03 20:47:49.848583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.848109 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-cb88ddr64"] Nov 03 20:47:49.923170 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:49.920312 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" podUID="a497eb34-2be2-44df-98c4-9b74da04329f" containerName="payload-processing" containerID="cri-o://8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde" gracePeriod=25 Nov 03 20:47:50.210369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.210339 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:47:50.332582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.332554 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s\") pod \"a497eb34-2be2-44df-98c4-9b74da04329f\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " Nov 03 20:47:50.333086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.332791 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials\") pod \"a497eb34-2be2-44df-98c4-9b74da04329f\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " Nov 03 20:47:50.333086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.332939 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume\") pod \"a497eb34-2be2-44df-98c4-9b74da04329f\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " Nov 03 20:47:50.333086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.332966 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume\") pod \"a497eb34-2be2-44df-98c4-9b74da04329f\" (UID: \"a497eb34-2be2-44df-98c4-9b74da04329f\") " Nov 03 20:47:50.333366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.333340 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume" pod "a497eb34-2be2-44df-98c4-9b74da04329f" (UID: "a497eb34-2be2-44df-98c4-9b74da04329f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:50.333467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.333409 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume" pod "a497eb34-2be2-44df-98c4-9b74da04329f" (UID: "a497eb34-2be2-44df-98c4-9b74da04329f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:50.333727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.333711 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:50.333771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.333731 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a497eb34-2be2-44df-98c4-9b74da04329f-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:50.335081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.335060 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s" pod "a497eb34-2be2-44df-98c4-9b74da04329f" (UID: "a497eb34-2be2-44df-98c4-9b74da04329f"). InnerVolumeSpecName "kube-api-access-pks8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:50.335227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.335213 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials" pod "a497eb34-2be2-44df-98c4-9b74da04329f" (UID: "a497eb34-2be2-44df-98c4-9b74da04329f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:50.435013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.434981 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:50.435013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.435009 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pks8s\" (UniqueName: \"kubernetes.io/projected/a497eb34-2be2-44df-98c4-9b74da04329f-kube-api-access-pks8s\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:50.629499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.629416 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b4bc6d-436c-4573-a82a-548f0fc55e48" path="/var/lib/kubelet/pods/50b4bc6d-436c-4573-a82a-548f0fc55e48/volumes" Nov 03 20:47:50.776079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.776035 5815 generic.go:356] "Generic (PLEG): container finished" podID="a497eb34-2be2-44df-98c4-9b74da04329f" containerID="8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde" exitCode=0 Nov 03 20:47:50.776289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.776133 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" Nov 03 20:47:50.776289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.776280 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" event={"ID":"a497eb34-2be2-44df-98c4-9b74da04329f","Type":"ContainerDied","Data":"8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde"} Nov 03 20:47:50.776428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.776330 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw" event={"ID":"a497eb34-2be2-44df-98c4-9b74da04329f","Type":"ContainerDied","Data":"c1e84e3b67b4ee62f80316941f2d715306b3ef2e66860011713c26c4e024959b"} Nov 03 20:47:50.776428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.776362 5815 scope.go:122] "RemoveContainer" containerID="8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde" Nov 03 20:47:50.810364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.810340 5815 scope.go:122] "RemoveContainer" containerID="8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde" Nov 03 20:47:50.810964 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:47:50.810933 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde\": container with ID starting with 8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde not found: ID does not exist" containerID="8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde" Nov 03 20:47:50.811099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.810975 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde"} err="failed to get container status \"8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde\": rpc error: code = NotFound desc = could not find container \"8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde\": container with ID starting with 8f0f2a86ce55fdd7ec0853c439f304f47ef117614571b3fd3d1b73f48ba52dde not found: ID does not exist" Nov 03 20:47:50.820592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.813193 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw"] Nov 03 20:47:50.824013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:50.823986 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-84cbwfgkw"] Nov 03 20:47:51.100018 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:51.099979 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:47:52.245674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.245642 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7"] Nov 03 20:47:52.246455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246439 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a497eb34-2be2-44df-98c4-9b74da04329f" containerName="payload-processing" Nov 03 20:47:52.246521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246456 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a497eb34-2be2-44df-98c4-9b74da04329f" containerName="payload-processing" Nov 03 20:47:52.246521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246481 5815 cpu_manager.go:414] "containerMap: removing container" podUID="50b4bc6d-436c-4573-a82a-548f0fc55e48" containerName="payload-processing" Nov 03 20:47:52.246521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246486 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b4bc6d-436c-4573-a82a-548f0fc55e48" containerName="payload-processing" Nov 03 20:47:52.246632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246551 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c03aec59-a797-46b1-a32b-48df42809845" containerName="revoke-keys" Nov 03 20:47:52.246632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246558 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03aec59-a797-46b1-a32b-48df42809845" containerName="revoke-keys" Nov 03 20:47:52.246695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246690 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03aec59-a797-46b1-a32b-48df42809845" containerName="revoke-keys" Nov 03 20:47:52.246729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246702 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a497eb34-2be2-44df-98c4-9b74da04329f" containerName="payload-processing" Nov 03 20:47:52.246729 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.246713 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b4bc6d-436c-4573-a82a-548f0fc55e48" containerName="payload-processing" Nov 03 20:47:52.251986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.251965 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.258815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.258614 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d7c7fd\"" Nov 03 20:47:52.262719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.262693 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7"] Nov 03 20:47:52.269278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.269258 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d7c7fd-dockercfg-7v2zx\"" Nov 03 20:47:52.313687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.313648 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q"] Nov 03 20:47:52.320589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.320561 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.340776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.340744 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q"] Nov 03 20:47:52.358765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.358720 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.359021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.358996 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42w6b\" (UniqueName: \"kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b\") pod \"payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.359125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.359036 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9t9p\" (UniqueName: \"kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p\") pod \"payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.359125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.359086 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.514581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.505343 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.514581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.505632 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42w6b\" (UniqueName: \"kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b\") pod \"payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.514581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.505667 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9t9p\" (UniqueName: \"kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p\") pod \"payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.514581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.505765 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.514581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.506877 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.577846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.577805 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.667889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.635474 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9t9p\" (UniqueName: \"kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p\") pod \"payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:52.729683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.718877 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42w6b\" (UniqueName: \"kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b\") pod \"payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.748242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.748201 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a497eb34-2be2-44df-98c4-9b74da04329f" path="/var/lib/kubelet/pods/a497eb34-2be2-44df-98c4-9b74da04329f/volumes" Nov 03 20:47:52.874247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.874160 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:52.970534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:52.966910 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:53.417141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.416079 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:53.486797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.485600 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:47:53.507302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.507255 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:53.680585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.677454 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:53.698556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.695577 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" probeResult="failure" output="Get \"https://10.132.1.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:53.748707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.747619 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:47:53.849534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.836909 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" probeResult="failure" output="Get \"https://10.132.1.110:8000/ready\": context deadline exceeded" Nov 03 20:47:53.980725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.980670 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:47:53.991198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:53.991158 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.148844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.147588 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" probeResult="failure" output="Get \"https://10.132.1.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.277331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.276933 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.417488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.417434 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.424432 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.418975 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:47:54.489674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.489612 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.490540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.490484 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:47:54.517647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.517595 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.518280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.518258 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:47:54.739646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.739602 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" probeResult="failure" output="Get \"https://10.132.1.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.763576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.756950 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/ready\": context deadline exceeded" Nov 03 20:47:54.764880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.764843 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:47:54.890747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.878256 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" probeResult="failure" output="Get \"https://10.132.1.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.994537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:54.982872 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:47:55.203176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.182463 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.295407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.182597 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" probeResult="failure" output="Get \"https://10.132.1.117:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.295407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.281824 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.448528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.445415 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.506533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.495719 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.573760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.573653 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.595789 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:47:55.595759 5815 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.008s" Nov 03 20:47:55.692923 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.596122 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:47:55.716667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.716636 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:47:55.717354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.717334 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:47:55.788009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.769209 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.819970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.819906 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:55.839668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.718877 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:47:55.841240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.841213 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:47:55.877195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.877161 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:47:55.877470 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.877451 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:47:55.877667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.877653 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:47:55.877832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.877818 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:47:55.877970 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:55.877959 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:47:56.249533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.230352 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:47:56.466728 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.466205 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" probeResult="failure" output="Get \"https://10.132.1.117:8000/ready\": context deadline exceeded" Nov 03 20:47:56.469531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.252123 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/health\": context deadline exceeded" Nov 03 20:47:56.601770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.588554 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf"] Nov 03 20:47:56.603465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.603400 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:56.635797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.635757 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:47:56.698008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.697969 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:56.743338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.738207 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:47:56.743338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.738340 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:47:56.743338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.738371 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:47:56.743338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.738402 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:47:56.743338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.738421 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7"] Nov 03 20:47:56.758153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.758075 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q"] Nov 03 20:47:56.888192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.888157 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" event={"ID":"7b8726ff-df3c-4b16-a272-5567d6f38b3a","Type":"ContainerStarted","Data":"cc8ffb1319354f472fa5ddb10a25d39a7135fcec80970a49a9e09b18209c0769"} Nov 03 20:47:56.888486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.888467 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:47:56.888683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.888666 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" event={"ID":"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac","Type":"ContainerStarted","Data":"2633281a9f6ff135a6e1ba31ae46cf2300fa2dbae68b9c3f505136f7531c904d"} Nov 03 20:47:56.888817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.888805 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:47:56.963926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:56.947915 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:47:57.913854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:57.913819 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" event={"ID":"7b8726ff-df3c-4b16-a272-5567d6f38b3a","Type":"ContainerStarted","Data":"784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962"} Nov 03 20:47:57.914422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:57.914025 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:47:57.916610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:57.916586 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" event={"ID":"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac","Type":"ContainerStarted","Data":"decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf"} Nov 03 20:47:57.916947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:57.916923 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:47:57.936804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:57.936763 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" podStartSLOduration=5.936750481 podStartE2EDuration="5.936750481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.93332039 +0000 UTC m=+2787.463666061" watchObservedRunningTime="2026-11-03 20:47:57.936750481 +0000 UTC m=+2787.467096153" Nov 03 20:47:57.957095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:47:57.957043 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" podStartSLOduration=5.957025318 podStartE2EDuration="5.957025318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:57.952259955 +0000 UTC m=+2787.482605627" watchObservedRunningTime="2026-11-03 20:47:57.957025318 +0000 UTC m=+2787.487370991" Nov 03 20:48:01.589790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:01.589717 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" podUID="8fdba816-d054-4202-befa-90955295a454" containerName="payload-processing" containerID="cri-o://0a102317872cb013dbd08480479ffe6ce90e713ff34361daa8acaea3be676c09" gracePeriod=25 Nov 03 20:48:01.955951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:01.955919 5815 generic.go:356] "Generic (PLEG): container finished" podID="8fdba816-d054-4202-befa-90955295a454" containerID="0a102317872cb013dbd08480479ffe6ce90e713ff34361daa8acaea3be676c09" exitCode=0 Nov 03 20:48:01.956092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:01.955999 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" event={"ID":"8fdba816-d054-4202-befa-90955295a454","Type":"ContainerDied","Data":"0a102317872cb013dbd08480479ffe6ce90e713ff34361daa8acaea3be676c09"} Nov 03 20:48:01.956092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:01.956066 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" event={"ID":"8fdba816-d054-4202-befa-90955295a454","Type":"ContainerDied","Data":"4d6530a0dd47e951722eccbb9aedc4b3b112de02885b3d8da371e66952e9aba9"} Nov 03 20:48:01.956196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:01.956092 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6530a0dd47e951722eccbb9aedc4b3b112de02885b3d8da371e66952e9aba9" Nov 03 20:48:01.972257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:01.972234 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:48:02.014225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.014195 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh\") pod \"8fdba816-d054-4202-befa-90955295a454\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " Nov 03 20:48:02.014339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.014272 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume\") pod \"8fdba816-d054-4202-befa-90955295a454\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " Nov 03 20:48:02.014495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.014479 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume\") pod \"8fdba816-d054-4202-befa-90955295a454\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " Nov 03 20:48:02.014665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.014641 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials\") pod \"8fdba816-d054-4202-befa-90955295a454\" (UID: \"8fdba816-d054-4202-befa-90955295a454\") " Nov 03 20:48:02.014780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.014635 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume" pod "8fdba816-d054-4202-befa-90955295a454" (UID: "8fdba816-d054-4202-befa-90955295a454"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:02.015023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.014932 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume" pod "8fdba816-d054-4202-befa-90955295a454" (UID: "8fdba816-d054-4202-befa-90955295a454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:02.015469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.015449 5815 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-routing-overlay-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:02.015629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.015471 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdba816-d054-4202-befa-90955295a454-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:02.016815 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.016796 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh" pod "8fdba816-d054-4202-befa-90955295a454" (UID: "8fdba816-d054-4202-befa-90955295a454"). InnerVolumeSpecName "kube-api-access-jqkmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:02.016910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.016824 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials" pod "8fdba816-d054-4202-befa-90955295a454" (UID: "8fdba816-d054-4202-befa-90955295a454"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:02.116668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.116641 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqkmh\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-kube-api-access-jqkmh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:02.116668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.116668 5815 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fdba816-d054-4202-befa-90955295a454-provider-credentials\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:02.605937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.605901 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8"] Nov 03 20:48:02.607211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.607186 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8fdba816-d054-4202-befa-90955295a454" containerName="payload-processing" Nov 03 20:48:02.607211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.607211 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdba816-d054-4202-befa-90955295a454" containerName="payload-processing" Nov 03 20:48:02.607571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.607485 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdba816-d054-4202-befa-90955295a454" containerName="payload-processing" Nov 03 20:48:02.633938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.633854 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8"] Nov 03 20:48:02.634110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.633977 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.723803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.723768 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d63cd-openshift-default-dockercfg-2j946\"" Nov 03 20:48:02.729916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.729882 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.729926 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730100 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730230 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730465 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730591 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730748 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730718 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730900 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhvd\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.730968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.730940 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.833776 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.833846 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.833915 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.834036 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhvd\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.834064 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.834224 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.834258 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.834368 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.834411 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.835020 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.837831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.835260 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.860578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.854884 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.860578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.855868 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.860578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.856768 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.885530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.874455 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhvd\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.885530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.883092 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.908535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.899828 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.908535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.901340 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:02.966533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:02.966219 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:03.024536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.023308 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf" Nov 03 20:48:03.143600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.136554 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf"] Nov 03 20:48:03.181531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.180756 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-fddbzv2gf"] Nov 03 20:48:03.559873 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.559840 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8"] Nov 03 20:48:03.568092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.568050 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:48:03.568213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.568123 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:48:03.568213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:03.568153 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:48:04.039322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:04.039277 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" event={"ID":"e6fc1210-f8a1-49e7-8302-c92c3832919d","Type":"ContainerStarted","Data":"5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9"} Nov 03 20:48:04.039838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:04.039351 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" event={"ID":"e6fc1210-f8a1-49e7-8302-c92c3832919d","Type":"ContainerStarted","Data":"00ff54c0a7ddd9a807f53ddea5259e48ff086905229383f351e09d45a7834332"} Nov 03 20:48:04.065890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:04.065831 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podStartSLOduration=2.065812758 podStartE2EDuration="2.065812758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:04.062288951 +0000 UTC m=+2793.592634624" watchObservedRunningTime="2026-11-03 20:48:04.065812758 +0000 UTC m=+2793.596158433" Nov 03 20:48:04.594861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:04.594824 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdba816-d054-4202-befa-90955295a454" path="/var/lib/kubelet/pods/8fdba816-d054-4202-befa-90955295a454/volumes" Nov 03 20:48:04.978529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:04.969333 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:04.986838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:04.986806 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:05.049697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:05.049665 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:05.050487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:05.050465 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:48:08.929817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:08.929735 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:48:08.929817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:08.929815 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:48:13.856084 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:48:13.856044 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:48:25.026224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:25.026182 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w"] Nov 03 20:48:25.031408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:25.031382 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3565bf14-kqf2w"] Nov 03 20:48:26.590349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:26.590313 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73eb9e7c-2ed0-4b75-b99b-b6630cd05925" path="/var/lib/kubelet/pods/73eb9e7c-2ed0-4b75-b99b-b6630cd05925/volumes" Nov 03 20:48:28.038261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:28.038224 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj"] Nov 03 20:48:28.044586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:28.044558 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e136d4b-9xfhj"] Nov 03 20:48:28.589882 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:28.589848 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ffdcfe-a821-4f1c-ab66-f35dddc89e28" path="/var/lib/kubelet/pods/86ffdcfe-a821-4f1c-ab66-f35dddc89e28/volumes" Nov 03 20:48:43.855644 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:48:43.855610 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:48:47.836072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.836040 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh"] Nov 03 20:48:47.853646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.853616 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.864343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.864306 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh"] Nov 03 20:48:47.866053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.866014 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bf7965\"/\"openshift-service-ca.crt\"" Nov 03 20:48:47.866375 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.866355 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bf7965\"/\"kube-root-ca.crt\"" Nov 03 20:48:47.888148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.888106 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7mmc\" (UniqueName: \"kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.888426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.888407 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.888647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.888632 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.888910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.888885 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.889053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.889030 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.889238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.889215 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.907655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.907609 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bf7965\"/\"test-model-bf7965-kserve-self-signed-certs\"" Nov 03 20:48:47.907847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.907825 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bf7965\"/\"default-dockercfg-n4rzb\"" Nov 03 20:48:47.991360 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.991313 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.991558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.991368 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.991558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.991454 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.991750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.991729 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7mmc\" (UniqueName: \"kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.991977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.991811 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.991977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.991892 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.992977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.992924 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.993102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.992999 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.993341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.993315 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.995280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.995255 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:47.997978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:47.997952 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:48.008384 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:48.008361 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7mmc\" (UniqueName: \"kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc\") pod \"test-model-bf7965-kserve-6fb5fbf5d9-4whzh\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:48.211621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:48.210126 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:48.851847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:48.851801 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh"] Nov 03 20:48:49.110574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:49.109352 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" event={"ID":"a0496e47-f1e4-4d10-b865-787fe8338fb7","Type":"ContainerStarted","Data":"d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265"} Nov 03 20:48:49.110574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:49.109405 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" event={"ID":"a0496e47-f1e4-4d10-b865-787fe8338fb7","Type":"ContainerStarted","Data":"e51704becbe841deba26530ae0252092b37038022ecb43cefa7cf76cc1e4e73f"} Nov 03 20:48:49.130252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:49.130192 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podStartSLOduration=2.13017945 podStartE2EDuration="2.13017945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:49.129577405 +0000 UTC m=+2838.659923078" watchObservedRunningTime="2026-11-03 20:48:49.13017945 +0000 UTC m=+2838.660525122" Nov 03 20:48:49.211261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:49.211215 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:50.805848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:50.805811 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:48:50.806579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:50.806473 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:48:53.244648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:53.244490 5815 generic.go:356] "Generic (PLEG): container finished" podID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerID="afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f" exitCode=0 Nov 03 20:48:53.245606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:53.244719 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" event={"ID":"904186e6-a013-4ce4-ab77-0d5893e2df5e","Type":"ContainerDied","Data":"afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f"} Nov 03 20:48:53.245606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:53.245317 5815 scope.go:122] "RemoveContainer" containerID="afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f" Nov 03 20:48:53.248424 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:53.248386 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:48:53.248647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:53.248488 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:48:53.248647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:53.248562 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:48:54.257119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:54.257083 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" event={"ID":"904186e6-a013-4ce4-ab77-0d5893e2df5e","Type":"ContainerStarted","Data":"16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7"} Nov 03 20:48:54.672587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:54.662833 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:48:54.694672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:54.694637 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:48:54.694956 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:54.694924 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:48:55.659142 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:55.659107 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:48:55.659674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:55.659195 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:48:56.658900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:56.658865 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:48:56.659194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:56.658928 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:48:57.659842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:57.659804 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:48:57.660263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:57.659886 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:48:58.214256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:58.214223 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:58.666642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:58.660733 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:48:58.666642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:58.660787 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:48:58.734967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:58.734938 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:58.747398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:58.747373 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:48:59.659395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:59.659359 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:48:59.659641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:48:59.659419 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:49:00.108712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.108643 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh"] Nov 03 20:49:00.125123 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.124980 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh"] Nov 03 20:49:00.125378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.125096 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.139276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.138993 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d7c7fd\"/\"test-model-d7c7fd-kserve-self-signed-certs\"" Nov 03 20:49:00.139276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.139078 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d7c7fd\"/\"default-dockercfg-l6hzw\"" Nov 03 20:49:00.139276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.139143 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d7c7fd\"/\"kube-root-ca.crt\"" Nov 03 20:49:00.139276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.139225 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d7c7fd\"/\"openshift-service-ca.crt\"" Nov 03 20:49:00.240469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.240416 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.240762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.240539 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.240762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.240583 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.240762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.240646 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxsvb\" (UniqueName: \"kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.240991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.240930 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.240991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.240966 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.353917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.353870 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.363723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.363625 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.364207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.364162 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.364460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.364439 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.364692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.364672 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.377529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.364855 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxsvb\" (UniqueName: \"kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.378050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.355496 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.378186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.371251 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.378282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.371886 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.431533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.428264 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxsvb\" (UniqueName: \"kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.445541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.434998 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.445541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.437284 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm\") pod \"test-model-d7c7fd-kserve-7d8d5b6645-c4mrh\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.462095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.461824 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:00.790529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.790222 5815 patch_prober.go:29] interesting pod/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" start-of-body= Nov 03 20:49:00.790529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:00.790297 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.127:15021/healthz/ready\": dial tcp 10.132.1.127:15021: connect: connection refused" Nov 03 20:49:01.137109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.136765 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh"] Nov 03 20:49:01.571957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.571919 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" event={"ID":"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c","Type":"ContainerStarted","Data":"64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a"} Nov 03 20:49:01.572167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.571970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" event={"ID":"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c","Type":"ContainerStarted","Data":"b56e9f177413f3bfa90a87a4b632ed903bab8a18137a9593d2d921707707b559"} Nov 03 20:49:01.596978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.596897 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podStartSLOduration=1.5968803409999999 podStartE2EDuration="1.596880341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:01.59148321 +0000 UTC m=+2851.121828882" watchObservedRunningTime="2026-11-03 20:49:01.596880341 +0000 UTC m=+2851.127226015" Nov 03 20:49:01.665774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.665740 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:49:01.666911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.666491 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:49:01.668262 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:01.668243 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:49:02.464579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:02.463330 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:10.463527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:10.463455 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:10.477936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:10.477908 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:10.739673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:10.739590 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:49:13.874128 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:49:13.874091 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:49:16.757921 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:16.757880 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj"] Nov 03 20:49:16.758419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:16.758100 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" containerID="cri-o://a91a2ed742393ea07b306dd840678d01ebf2f778ce88a0ee494cf1553823e422" gracePeriod=30 Nov 03 20:49:21.499633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:21.499594 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-k6ntj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:49:21.500049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:21.499673 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:49:22.009586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.009215 5815 generic.go:356] "Generic (PLEG): container finished" podID="51daf21c-856d-4f43-ba29-77e134c1b464" containerID="a91a2ed742393ea07b306dd840678d01ebf2f778ce88a0ee494cf1553823e422" exitCode=0 Nov 03 20:49:22.009586 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.009490 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" event={"ID":"51daf21c-856d-4f43-ba29-77e134c1b464","Type":"ContainerDied","Data":"a91a2ed742393ea07b306dd840678d01ebf2f778ce88a0ee494cf1553823e422"} Nov 03 20:49:22.071500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.071470 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:49:22.107409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107374 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.107631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107459 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.107631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107487 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.107631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107619 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107657 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107747 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107747 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107847 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.107951 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.108080 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.108131 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs\") pod \"51daf21c-856d-4f43-ba29-77e134c1b464\" (UID: \"51daf21c-856d-4f43-ba29-77e134c1b464\") " Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.108280 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.108367 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:22.108796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.108605 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:22.109406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.109370 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.109498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.109413 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.109498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.109423 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.109498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.109432 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.109498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.109440 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51daf21c-856d-4f43-ba29-77e134c1b464-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.113408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.113362 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:49:22.113560 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.113425 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "kube-api-access-9wvq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:22.113630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.113587 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:22.113682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.113633 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token" pod "51daf21c-856d-4f43-ba29-77e134c1b464" (UID: "51daf21c-856d-4f43-ba29-77e134c1b464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:22.221959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.221924 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51daf21c-856d-4f43-ba29-77e134c1b464-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.221959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.221959 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51daf21c-856d-4f43-ba29-77e134c1b464-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.222248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.221979 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:22.222248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:22.222001 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvq8\" (UniqueName: \"kubernetes.io/projected/51daf21c-856d-4f43-ba29-77e134c1b464-kube-api-access-9wvq8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:23.085832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:23.076327 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" event={"ID":"51daf21c-856d-4f43-ba29-77e134c1b464","Type":"ContainerDied","Data":"c52319752f0096e485223e1419d70e5e2d3643c60adbe9d0dfe9cbf6c7fef03f"} Nov 03 20:49:23.085832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:23.076478 5815 scope.go:122] "RemoveContainer" containerID="a91a2ed742393ea07b306dd840678d01ebf2f778ce88a0ee494cf1553823e422" Nov 03 20:49:23.085832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:23.076823 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj" Nov 03 20:49:23.112500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:23.112159 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj"] Nov 03 20:49:23.117311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:23.117284 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-k6ntj"] Nov 03 20:49:24.595152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:24.595109 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" path="/var/lib/kubelet/pods/51daf21c-856d-4f43-ba29-77e134c1b464/volumes" Nov 03 20:49:25.325715 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:25.325675 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:49:36.114675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:36.114635 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:49:36.115233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:36.114707 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:49:39.509897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.509864 5815 generic.go:356] "Generic (PLEG): container finished" podID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerID="b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9" exitCode=0 Nov 03 20:49:39.510488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.509937 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" event={"ID":"6a47b882-1e6c-4aef-8b71-9e1b8629b857","Type":"ContainerDied","Data":"b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9"} Nov 03 20:49:39.510656 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.510637 5815 scope.go:122] "RemoveContainer" containerID="b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9" Nov 03 20:49:39.512138 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.512115 5815 generic.go:356] "Generic (PLEG): container finished" podID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerID="5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9" exitCode=0 Nov 03 20:49:39.512240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.512163 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" event={"ID":"e6fc1210-f8a1-49e7-8302-c92c3832919d","Type":"ContainerDied","Data":"5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9"} Nov 03 20:49:39.512623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.512608 5815 scope.go:122] "RemoveContainer" containerID="5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9" Nov 03 20:49:39.513318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.513194 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:49:39.513318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.513265 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:49:39.513407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.513318 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:49:39.514787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.514758 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:49:39.514917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.514837 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:49:39.514917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:39.514885 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:49:40.543883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.543850 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" event={"ID":"6a47b882-1e6c-4aef-8b71-9e1b8629b857","Type":"ContainerStarted","Data":"7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307"} Nov 03 20:49:40.546793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.546763 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" event={"ID":"e6fc1210-f8a1-49e7-8302-c92c3832919d","Type":"ContainerStarted","Data":"04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95"} Nov 03 20:49:40.949579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.948867 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:49:40.955531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.954641 5815 patch_prober.go:29] interesting pod/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:49:40.955531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.954708 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:49:40.978787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.975898 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:49:40.978787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.976077 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:49:40.978787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:40.976121 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:49:41.944531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:41.944151 5815 patch_prober.go:29] interesting pod/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:49:41.944531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:41.944207 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:49:41.987532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:41.986991 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:49:41.987532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:41.987050 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:49:42.943854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:42.943811 5815 patch_prober.go:29] interesting pod/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:49:42.944066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:42.943878 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:49:42.981532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:42.968840 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:49:42.981532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:42.968894 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:49:43.733446 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:43.733392 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:49:43.945530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:43.943888 5815 patch_prober.go:29] interesting pod/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:49:43.945530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:43.943956 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:49:43.985613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:43.985493 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:49:43.986197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:43.986165 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:49:44.036817 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:49:44.036761 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:49:44.944009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:44.943975 5815 patch_prober.go:29] interesting pod/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:49:44.944208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:44.944021 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:49:44.988534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:44.988051 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:49:44.988534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:44.988137 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:49:45.562289 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:45.562242 5815 scope.go:122] "RemoveContainer" containerID="e3a317220a3a7ab7c4011c1c752ceba7e275e695755835cbd7db4730e736d533" Nov 03 20:49:45.583458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:45.583436 5815 scope.go:122] "RemoveContainer" containerID="d41a30bfb4a273d5bde0800200fc647179a1d329b34dcb0aaa28d0c14b9c23c4" Nov 03 20:49:45.944291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:45.944210 5815 patch_prober.go:29] interesting pod/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" start-of-body= Nov 03 20:49:45.944291 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:45.944265 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.121:15021/healthz/ready\": dial tcp 10.132.1.121:15021: connect: connection refused" Nov 03 20:49:45.974530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:45.972811 5815 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" start-of-body= Nov 03 20:49:45.974530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:45.972869 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.130:15021/healthz/ready\": dial tcp 10.132.1.130:15021: connect: connection refused" Nov 03 20:49:46.949066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:46.949031 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:49:46.950067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:46.950042 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:49:46.972481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:46.972458 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:49:46.972770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:46.972750 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:49:46.973577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:46.973556 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:49:53.993675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:53.992426 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk"] Nov 03 20:49:53.994161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:53.993754 5815 cpu_manager.go:414] "containerMap: removing container" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" Nov 03 20:49:53.994161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:53.993776 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" Nov 03 20:49:53.994161 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:53.994065 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="51daf21c-856d-4f43-ba29-77e134c1b464" containerName="istio-proxy" Nov 03 20:49:54.022589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.016940 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.022589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.022182 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk"] Nov 03 20:49:54.023004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.022983 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d63cd\"/\"default-dockercfg-7ng4z\"" Nov 03 20:49:54.023122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.023031 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d63cd\"/\"kube-root-ca.crt\"" Nov 03 20:49:54.023122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.023088 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d63cd\"/\"test-model-4d63cd-kserve-self-signed-certs\"" Nov 03 20:49:54.023243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.023175 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d63cd\"/\"openshift-service-ca.crt\"" Nov 03 20:49:54.065714 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.065671 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.065893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.065748 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.066084 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.066067 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.066146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.066100 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.066204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.066147 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.066204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.066192 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznc8\" (UniqueName: \"kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.170535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.170094 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.170535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.170200 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.170535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.170339 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.170535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.170412 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.187536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.181998 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.187536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.182067 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.187536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.182185 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznc8\" (UniqueName: \"kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.187536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.183110 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.187536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.183342 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.292582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.278108 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.316529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.303088 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.333533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.332443 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznc8\" (UniqueName: \"kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8\") pod \"test-model-4d63cd-kserve-8455dfbd5f-htfwk\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.343530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.338816 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:49:54.856257 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:49:54.856221 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90948292_dd9b_4b36_8608_000229f5c249.slice/crio-39d3e1ccd7d3e517d5f0ebd97a6f7254e5b5f9f25ff9b70d8034df1997deff5f WatchSource:0}: Error finding container 39d3e1ccd7d3e517d5f0ebd97a6f7254e5b5f9f25ff9b70d8034df1997deff5f: Status 404 returned error can't find the container with id 39d3e1ccd7d3e517d5f0ebd97a6f7254e5b5f9f25ff9b70d8034df1997deff5f Nov 03 20:49:54.856629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.856606 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk"] Nov 03 20:49:54.957131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:54.957096 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" event={"ID":"90948292-dd9b-4b36-8608-000229f5c249","Type":"ContainerStarted","Data":"39d3e1ccd7d3e517d5f0ebd97a6f7254e5b5f9f25ff9b70d8034df1997deff5f"} Nov 03 20:49:55.969302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:55.969260 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" event={"ID":"90948292-dd9b-4b36-8608-000229f5c249","Type":"ContainerStarted","Data":"7df99da29895a4a47d47ca139832fd077632a8db9dff3d83456faf9617a09e44"} Nov 03 20:49:55.990785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:55.990726 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" podStartSLOduration=2.99070695 podStartE2EDuration="2.99070695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:55.987897507 +0000 UTC m=+2905.518243180" watchObservedRunningTime="2026-11-03 20:49:55.99070695 +0000 UTC m=+2905.521052625" Nov 03 20:49:56.341406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:49:56.341331 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:50:04.339770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:04.339733 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:50:04.353368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:04.353342 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:50:05.092865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:05.092836 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:50:06.624308 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.624275 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9"] Nov 03 20:50:06.633286 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.633260 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.638479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.638450 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9"] Nov 03 20:50:06.641263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.641243 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-519a78\"/\"default-dockercfg-t7x5n\"" Nov 03 20:50:06.641639 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.641492 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-519a78\"/\"kube-root-ca.crt\"" Nov 03 20:50:06.643356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.643338 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-519a78\"/\"openshift-service-ca.crt\"" Nov 03 20:50:06.643695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.643647 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-519a78\"/\"test-model-519a78-kserve-self-signed-certs\"" Nov 03 20:50:06.756358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.756317 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.756705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.756681 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.757245 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.757212 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6h8m\" (UniqueName: \"kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.757530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.757487 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.757704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.757687 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.757903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.757885 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.917618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.872866 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.918133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.918100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6h8m\" (UniqueName: \"kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.918373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.918355 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.918548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.918531 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.918764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.918748 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.918937 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.874963 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.919173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.919154 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.939536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.922734 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.939776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.937183 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.966535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.946618 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6h8m\" (UniqueName: \"kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.966835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.953132 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:06.973007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:06.972976 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm\") pod \"test-model-519a78-kserve-bfcf7fc57-vk4x9\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:07.256887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:07.256848 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:07.636468 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:50:07.636440 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc34b18ef_3f1a_4291_849f_8a297b233b93.slice/crio-86979c7678906672b9f7356455554d18226a5a2ba44f47dd00535f1aef0d2c36 WatchSource:0}: Error finding container 86979c7678906672b9f7356455554d18226a5a2ba44f47dd00535f1aef0d2c36: Status 404 returned error can't find the container with id 86979c7678906672b9f7356455554d18226a5a2ba44f47dd00535f1aef0d2c36 Nov 03 20:50:07.636938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:07.636731 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9"] Nov 03 20:50:08.154467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:08.154435 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" event={"ID":"c34b18ef-3f1a-4291-849f-8a297b233b93","Type":"ContainerStarted","Data":"0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87"} Nov 03 20:50:08.154666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:08.154484 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" event={"ID":"c34b18ef-3f1a-4291-849f-8a297b233b93","Type":"ContainerStarted","Data":"86979c7678906672b9f7356455554d18226a5a2ba44f47dd00535f1aef0d2c36"} Nov 03 20:50:08.182781 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:08.182723 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" podStartSLOduration=2.182705486 podStartE2EDuration="2.182705486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:08.174940338 +0000 UTC m=+2917.705286010" watchObservedRunningTime="2026-11-03 20:50:08.182705486 +0000 UTC m=+2917.713051159" Nov 03 20:50:08.262140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:08.262112 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:13.854065 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:50:13.854032 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:50:17.262023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:17.261985 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:17.275315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:17.275287 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:17.323054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:17.323023 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:50:30.197654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.197617 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf"] Nov 03 20:50:30.202576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.202556 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.210695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.210672 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:30.219922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.212942 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf"] Nov 03 20:50:30.219922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.214655 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:50:30.252631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.252601 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.252778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.252646 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d76f\" (UniqueName: \"kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.253152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.253124 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.354317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.354286 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.354497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.354385 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.354497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.354414 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d76f\" (UniqueName: \"kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.355079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.355059 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.357036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.357018 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.366877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.366850 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d76f\" (UniqueName: \"kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f\") pod \"maas-api-revoke-sub-cb16cb06-qjwnf\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.525836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.525806 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:30.844182 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:50:30.844155 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fda728_89ef_4bf5_ad38_55fcbc847d2a.slice/crio-b33aeca819c33b8ba618c02746b22c1154a2b1e7ebe940fd25c78e84f2c928c5 WatchSource:0}: Error finding container b33aeca819c33b8ba618c02746b22c1154a2b1e7ebe940fd25c78e84f2c928c5: Status 404 returned error can't find the container with id b33aeca819c33b8ba618c02746b22c1154a2b1e7ebe940fd25c78e84f2c928c5 Nov 03 20:50:30.845821 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:30.845792 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf"] Nov 03 20:50:31.607316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:31.606023 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" event={"ID":"94fda728-89ef-4bf5-ad38-55fcbc847d2a","Type":"ContainerStarted","Data":"7f408ee071a768c9ac40dfecbade0a28859bd1269ecc58f59d3e3363c45b5c99"} Nov 03 20:50:31.607316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:31.606097 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" event={"ID":"94fda728-89ef-4bf5-ad38-55fcbc847d2a","Type":"ContainerStarted","Data":"b33aeca819c33b8ba618c02746b22c1154a2b1e7ebe940fd25c78e84f2c928c5"} Nov 03 20:50:31.630871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:31.630816 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" podStartSLOduration=1.630800792 podStartE2EDuration="1.630800792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:31.624983952 +0000 UTC m=+2941.155329625" watchObservedRunningTime="2026-11-03 20:50:31.630800792 +0000 UTC m=+2941.161146473" Nov 03 20:50:32.618908 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:32.618874 5815 generic.go:356] "Generic (PLEG): container finished" podID="94fda728-89ef-4bf5-ad38-55fcbc847d2a" containerID="7f408ee071a768c9ac40dfecbade0a28859bd1269ecc58f59d3e3363c45b5c99" exitCode=0 Nov 03 20:50:32.619353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:32.618950 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" event={"ID":"94fda728-89ef-4bf5-ad38-55fcbc847d2a","Type":"ContainerDied","Data":"7f408ee071a768c9ac40dfecbade0a28859bd1269ecc58f59d3e3363c45b5c99"} Nov 03 20:50:33.025832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:33.025800 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg"] Nov 03 20:50:33.031945 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:33.031906 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d046963a-4jxtg"] Nov 03 20:50:33.944393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:33.944364 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:34.014939 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.014879 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f\" (UniqueName: \"kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f\") pod \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " Nov 03 20:50:34.015585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.015561 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca\") pod \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " Nov 03 20:50:34.015780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.015764 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca\") pod \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\" (UID: \"94fda728-89ef-4bf5-ad38-55fcbc847d2a\") " Nov 03 20:50:34.016285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.015924 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca" pod "94fda728-89ef-4bf5-ad38-55fcbc847d2a" (UID: "94fda728-89ef-4bf5-ad38-55fcbc847d2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:34.017331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.017300 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:34.019126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.019095 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f" pod "94fda728-89ef-4bf5-ad38-55fcbc847d2a" (UID: "94fda728-89ef-4bf5-ad38-55fcbc847d2a"). InnerVolumeSpecName "kube-api-access-7d76f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:34.019419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.019398 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca" pod "94fda728-89ef-4bf5-ad38-55fcbc847d2a" (UID: "94fda728-89ef-4bf5-ad38-55fcbc847d2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:34.119158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.119128 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d76f\" (UniqueName: \"kubernetes.io/projected/94fda728-89ef-4bf5-ad38-55fcbc847d2a-kube-api-access-7d76f\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:34.119158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.119158 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fda728-89ef-4bf5-ad38-55fcbc847d2a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:34.596153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.596116 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae379051-3d3d-4dc5-8f68-79dba06cb76f" path="/var/lib/kubelet/pods/ae379051-3d3d-4dc5-8f68-79dba06cb76f/volumes" Nov 03 20:50:34.639773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.639739 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" event={"ID":"94fda728-89ef-4bf5-ad38-55fcbc847d2a","Type":"ContainerDied","Data":"b33aeca819c33b8ba618c02746b22c1154a2b1e7ebe940fd25c78e84f2c928c5"} Nov 03 20:50:34.639773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.639765 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf" Nov 03 20:50:34.640011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:34.639795 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b33aeca819c33b8ba618c02746b22c1154a2b1e7ebe940fd25c78e84f2c928c5" Nov 03 20:50:40.807581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:40.805895 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft"] Nov 03 20:50:41.293163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.293130 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr"] Nov 03 20:50:41.294045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.294027 5815 cpu_manager.go:414] "containerMap: removing container" podUID="94fda728-89ef-4bf5-ad38-55fcbc847d2a" containerName="revoke-keys" Nov 03 20:50:41.294045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.294046 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fda728-89ef-4bf5-ad38-55fcbc847d2a" containerName="revoke-keys" Nov 03 20:50:41.294235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.294225 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fda728-89ef-4bf5-ad38-55fcbc847d2a" containerName="revoke-keys" Nov 03 20:50:41.299840 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.299820 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.305188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.305135 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:41.305321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.305209 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:50:41.311558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.310559 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr"] Nov 03 20:50:41.437578 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.437494 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6g57\" (UniqueName: \"kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.437763 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.437712 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.437848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.437825 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.539473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.539433 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.539721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.539705 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6g57\" (UniqueName: \"kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.539822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.539808 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.540307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.540281 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.542755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.542726 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.552328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.552269 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6g57\" (UniqueName: \"kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57\") pod \"maas-api-revoke-sub-8ec8b2d0-t6wxr\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.623981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.623947 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:41.923334 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:50:41.923302 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e385ddf_104c_46f9_8aea_0fbb0aa782fb.slice/crio-776dd92219eebf0f443596c8658e3972675b8ed8817397efad7b9bf661fbbd77 WatchSource:0}: Error finding container 776dd92219eebf0f443596c8658e3972675b8ed8817397efad7b9bf661fbbd77: Status 404 returned error can't find the container with id 776dd92219eebf0f443596c8658e3972675b8ed8817397efad7b9bf661fbbd77 Nov 03 20:50:41.924677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:41.924647 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr"] Nov 03 20:50:42.025897 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.025866 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k"] Nov 03 20:50:42.030924 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.030900 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e58d5ff0-jzr5k"] Nov 03 20:50:42.609831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.609797 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd7d3eb-fca3-4858-97ce-f6aafc487f25" path="/var/lib/kubelet/pods/efd7d3eb-fca3-4858-97ce-f6aafc487f25/volumes" Nov 03 20:50:42.826089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.826041 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" event={"ID":"1e385ddf-104c-46f9-8aea-0fbb0aa782fb","Type":"ContainerStarted","Data":"b05a56f9c86f3b31500b1d43f03bd2bcc9106579bc389e36da2f6fec508b1a4b"} Nov 03 20:50:42.826276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.826117 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" event={"ID":"1e385ddf-104c-46f9-8aea-0fbb0aa782fb","Type":"ContainerStarted","Data":"776dd92219eebf0f443596c8658e3972675b8ed8817397efad7b9bf661fbbd77"} Nov 03 20:50:42.848405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.848349 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" podStartSLOduration=1.848336064 podStartE2EDuration="1.848336064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:42.845306178 +0000 UTC m=+2952.375651847" watchObservedRunningTime="2026-11-03 20:50:42.848336064 +0000 UTC m=+2952.378681742" Nov 03 20:50:42.987272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:42.987237 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:50:44.103532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:44.102444 5815 generic.go:356] "Generic (PLEG): container finished" podID="1e385ddf-104c-46f9-8aea-0fbb0aa782fb" containerID="b05a56f9c86f3b31500b1d43f03bd2bcc9106579bc389e36da2f6fec508b1a4b" exitCode=0 Nov 03 20:50:44.103532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:44.102537 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" event={"ID":"1e385ddf-104c-46f9-8aea-0fbb0aa782fb","Type":"ContainerDied","Data":"b05a56f9c86f3b31500b1d43f03bd2bcc9106579bc389e36da2f6fec508b1a4b"} Nov 03 20:50:44.178018 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:50:44.177109 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:50:45.669697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.669597 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:50:45.693048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.687775 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" probeResult="failure" output="Get \"https://10.132.1.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:45.746546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.743596 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.132:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:45.762868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.757933 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:45.797889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.788432 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:45.830844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.830800 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:45.832194 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.832159 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" probeResult="failure" output="Get \"https://10.132.1.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:45.832830 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.832806 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:50:45.887637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:45.886876 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:46.017672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.017368 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" probeResult="failure" output="Get \"https://10.132.1.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:46.142671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.142634 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:46.272371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.272281 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57\" (UniqueName: \"kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57\") pod \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " Nov 03 20:50:46.287791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.287744 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca" pod "1e385ddf-104c-46f9-8aea-0fbb0aa782fb" (UID: "1e385ddf-104c-46f9-8aea-0fbb0aa782fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:46.288191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.288171 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca\") pod \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " Nov 03 20:50:46.291808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.291787 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca\") pod \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\" (UID: \"1e385ddf-104c-46f9-8aea-0fbb0aa782fb\") " Nov 03 20:50:46.313977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.313950 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:46.340608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.337694 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57" pod "1e385ddf-104c-46f9-8aea-0fbb0aa782fb" (UID: "1e385ddf-104c-46f9-8aea-0fbb0aa782fb"). InnerVolumeSpecName "kube-api-access-z6g57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:46.340608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.337710 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca" pod "1e385ddf-104c-46f9-8aea-0fbb0aa782fb" (UID: "1e385ddf-104c-46f9-8aea-0fbb0aa782fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:46.355028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.355001 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" Nov 03 20:50:46.355635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.354996 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr" event={"ID":"1e385ddf-104c-46f9-8aea-0fbb0aa782fb","Type":"ContainerDied","Data":"776dd92219eebf0f443596c8658e3972675b8ed8817397efad7b9bf661fbbd77"} Nov 03 20:50:46.355731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.355668 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="776dd92219eebf0f443596c8658e3972675b8ed8817397efad7b9bf661fbbd77" Nov 03 20:50:46.417102 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.417068 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:46.417271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:46.417106 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6g57\" (UniqueName: \"kubernetes.io/projected/1e385ddf-104c-46f9-8aea-0fbb0aa782fb-kube-api-access-z6g57\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:48.656790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.656754 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw"] Nov 03 20:50:48.657636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.657618 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1e385ddf-104c-46f9-8aea-0fbb0aa782fb" containerName="revoke-keys" Nov 03 20:50:48.657735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.657639 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e385ddf-104c-46f9-8aea-0fbb0aa782fb" containerName="revoke-keys" Nov 03 20:50:48.657811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.657801 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e385ddf-104c-46f9-8aea-0fbb0aa782fb" containerName="revoke-keys" Nov 03 20:50:48.662156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.662134 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.666524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.666488 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:48.667334 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.667311 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:50:48.679024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.678948 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw"] Nov 03 20:50:48.763236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.763191 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.763426 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.763329 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.763619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.763602 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jn8b\" (UniqueName: \"kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.865525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.865470 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.865720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.865616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.865917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.865897 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn8b\" (UniqueName: \"kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.866274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.866249 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.868163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.868139 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.878596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.878573 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jn8b\" (UniqueName: \"kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b\") pod \"maas-api-revoke-sub-fb18a7da-wsslw\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:48.986695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:48.986671 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:49.398746 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:50:49.398715 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6febb5d1_e3ab_4d0e_911e_1aca297896d4.slice/crio-0a600602767060ca91bc03b6a14002b47464f486bba65746d94a7e5bd408d9ff WatchSource:0}: Error finding container 0a600602767060ca91bc03b6a14002b47464f486bba65746d94a7e5bd408d9ff: Status 404 returned error can't find the container with id 0a600602767060ca91bc03b6a14002b47464f486bba65746d94a7e5bd408d9ff Nov 03 20:50:49.398888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:49.398838 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw"] Nov 03 20:50:50.416524 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:50.416474 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" event={"ID":"6febb5d1-e3ab-4d0e-911e-1aca297896d4","Type":"ContainerStarted","Data":"1cc4df764316191e85cc7e0ff0e8b935c28c2fbbdc18fc23af57b7cb85ba33f7"} Nov 03 20:50:50.416952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:50.416583 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" event={"ID":"6febb5d1-e3ab-4d0e-911e-1aca297896d4","Type":"ContainerStarted","Data":"0a600602767060ca91bc03b6a14002b47464f486bba65746d94a7e5bd408d9ff"} Nov 03 20:50:50.437759 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:50.437701 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" podStartSLOduration=2.437681151 podStartE2EDuration="2.437681151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:50.434452371 +0000 UTC m=+2959.964798067" watchObservedRunningTime="2026-11-03 20:50:50.437681151 +0000 UTC m=+2959.968026826" Nov 03 20:50:51.433564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:51.433210 5815 generic.go:356] "Generic (PLEG): container finished" podID="6febb5d1-e3ab-4d0e-911e-1aca297896d4" containerID="1cc4df764316191e85cc7e0ff0e8b935c28c2fbbdc18fc23af57b7cb85ba33f7" exitCode=0 Nov 03 20:50:51.433564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:51.433290 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" event={"ID":"6febb5d1-e3ab-4d0e-911e-1aca297896d4","Type":"ContainerDied","Data":"1cc4df764316191e85cc7e0ff0e8b935c28c2fbbdc18fc23af57b7cb85ba33f7"} Nov 03 20:50:52.335889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:52.334637 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr"] Nov 03 20:50:52.984695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:52.984666 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:53.086575 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.055444 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca\") pod \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " Nov 03 20:50:53.097990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.088480 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca" pod "6febb5d1-e3ab-4d0e-911e-1aca297896d4" (UID: "6febb5d1-e3ab-4d0e-911e-1aca297896d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:53.098228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.098202 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca\") pod \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " Nov 03 20:50:53.098323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.098313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b\" (UniqueName: \"kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b\") pod \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\" (UID: \"6febb5d1-e3ab-4d0e-911e-1aca297896d4\") " Nov 03 20:50:53.099709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.099683 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:53.100491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.100464 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca" pod "6febb5d1-e3ab-4d0e-911e-1aca297896d4" (UID: "6febb5d1-e3ab-4d0e-911e-1aca297896d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:53.103316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.103271 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b" pod "6febb5d1-e3ab-4d0e-911e-1aca297896d4" (UID: "6febb5d1-e3ab-4d0e-911e-1aca297896d4"). InnerVolumeSpecName "kube-api-access-2jn8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:53.201294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.201222 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jn8b\" (UniqueName: \"kubernetes.io/projected/6febb5d1-e3ab-4d0e-911e-1aca297896d4-kube-api-access-2jn8b\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:53.201294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.201251 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6febb5d1-e3ab-4d0e-911e-1aca297896d4-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:53.676370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.676341 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" Nov 03 20:50:53.676599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.676368 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw" event={"ID":"6febb5d1-e3ab-4d0e-911e-1aca297896d4","Type":"ContainerDied","Data":"0a600602767060ca91bc03b6a14002b47464f486bba65746d94a7e5bd408d9ff"} Nov 03 20:50:53.676599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:53.676428 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a600602767060ca91bc03b6a14002b47464f486bba65746d94a7e5bd408d9ff" Nov 03 20:50:55.845485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:55.845428 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" containerID="cri-o://8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6" gracePeriod=45 Nov 03 20:50:56.264612 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.264588 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:50:56.290534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.290480 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home\") pod \"0c5c598f-2394-40f5-95e1-42c38a587203\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " Nov 03 20:50:56.290720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.290543 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs\") pod \"0c5c598f-2394-40f5-95e1-42c38a587203\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " Nov 03 20:50:56.290720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.290616 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm\") pod \"0c5c598f-2394-40f5-95e1-42c38a587203\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " Nov 03 20:50:56.290720 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.290655 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache\") pod \"0c5c598f-2394-40f5-95e1-42c38a587203\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " Nov 03 20:50:56.292140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.291909 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home" pod "0c5c598f-2394-40f5-95e1-42c38a587203" (UID: "0c5c598f-2394-40f5-95e1-42c38a587203"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:56.292268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.292169 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache" pod "0c5c598f-2394-40f5-95e1-42c38a587203" (UID: "0c5c598f-2394-40f5-95e1-42c38a587203"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:56.292359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.292342 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs\" (UniqueName: \"kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs\") pod \"0c5c598f-2394-40f5-95e1-42c38a587203\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " Nov 03 20:50:56.292473 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.292418 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir\") pod \"0c5c598f-2394-40f5-95e1-42c38a587203\" (UID: \"0c5c598f-2394-40f5-95e1-42c38a587203\") " Nov 03 20:50:56.292902 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.292878 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir" pod "0c5c598f-2394-40f5-95e1-42c38a587203" (UID: "0c5c598f-2394-40f5-95e1-42c38a587203"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:56.293695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.293671 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:56.293933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.293698 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:56.293933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.293735 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:56.295698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.295675 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm" pod "0c5c598f-2394-40f5-95e1-42c38a587203" (UID: "0c5c598f-2394-40f5-95e1-42c38a587203"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:56.297268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.297248 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs" pod "0c5c598f-2394-40f5-95e1-42c38a587203" (UID: "0c5c598f-2394-40f5-95e1-42c38a587203"). InnerVolumeSpecName "kube-api-access-qrshs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:56.323658 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.323630 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs" pod "0c5c598f-2394-40f5-95e1-42c38a587203" (UID: "0c5c598f-2394-40f5-95e1-42c38a587203"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:56.395917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.395892 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5c598f-2394-40f5-95e1-42c38a587203-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:56.395917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.395918 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c5c598f-2394-40f5-95e1-42c38a587203-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:56.396121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.395928 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrshs\" (UniqueName: \"kubernetes.io/projected/0c5c598f-2394-40f5-95e1-42c38a587203-kube-api-access-qrshs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:56.712755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.712724 5815 generic.go:356] "Generic (PLEG): container finished" podID="0c5c598f-2394-40f5-95e1-42c38a587203" containerID="8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6" exitCode=0 Nov 03 20:50:56.712929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.712778 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" Nov 03 20:50:56.712929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.712801 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" event={"ID":"0c5c598f-2394-40f5-95e1-42c38a587203","Type":"ContainerDied","Data":"8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6"} Nov 03 20:50:56.712929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.712845 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft" event={"ID":"0c5c598f-2394-40f5-95e1-42c38a587203","Type":"ContainerDied","Data":"9378fbff0840b8615a49f5c98a891004b563d8867369de8b49b9642904cd6920"} Nov 03 20:50:56.712929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.712857 5815 scope.go:122] "RemoveContainer" containerID="8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6" Nov 03 20:50:56.725328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.725309 5815 scope.go:122] "RemoveContainer" containerID="8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6" Nov 03 20:50:56.725670 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:50:56.725642 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6\": container with ID starting with 8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6 not found: ID does not exist" containerID="8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6" Nov 03 20:50:56.725787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.725676 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6"} err="failed to get container status \"8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6\": rpc error: code = NotFound desc = could not find container \"8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6\": container with ID starting with 8b597076e84048ca1a4f485370dc20d19a71ce2b74165e9b64b0c92e844f93c6 not found: ID does not exist" Nov 03 20:50:56.735236 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.735211 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft"] Nov 03 20:50:56.740796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:56.740773 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3d4fe2/test-model-3d4fe2-kserve-7fc8f74dc9-8dcft"] Nov 03 20:50:58.120849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:58.120811 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm"] Nov 03 20:50:58.132273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:58.132242 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj"] Nov 03 20:50:58.638798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:58.638762 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" path="/var/lib/kubelet/pods/0c5c598f-2394-40f5-95e1-42c38a587203/volumes" Nov 03 20:50:59.228744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.228705 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc"] Nov 03 20:50:59.229680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.229663 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6febb5d1-e3ab-4d0e-911e-1aca297896d4" containerName="revoke-keys" Nov 03 20:50:59.229737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.229681 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6febb5d1-e3ab-4d0e-911e-1aca297896d4" containerName="revoke-keys" Nov 03 20:50:59.229737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.229692 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" Nov 03 20:50:59.229737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.229697 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" Nov 03 20:50:59.229906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.229851 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5c598f-2394-40f5-95e1-42c38a587203" containerName="main" Nov 03 20:50:59.229906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.229866 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6febb5d1-e3ab-4d0e-911e-1aca297896d4" containerName="revoke-keys" Nov 03 20:50:59.237055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.237025 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.244131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.244104 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:59.245198 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.245170 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:50:59.249146 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.249116 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc"] Nov 03 20:50:59.342877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.342834 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.343075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.343058 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.343223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.343203 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rz7t\" (UniqueName: \"kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.444456 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.444425 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.444657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.444571 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rz7t\" (UniqueName: \"kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.444755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.444742 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.445347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.445320 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.447302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.447270 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.476074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.476044 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rz7t\" (UniqueName: \"kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t\") pod \"maas-api-revoke-sub-18024fb6-ffmbc\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.574494 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.574416 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:50:59.871862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:50:59.871830 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc"] Nov 03 20:51:00.832214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:00.832177 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" event={"ID":"6def77c8-4ce4-4bc5-af6c-599dfe585a00","Type":"ContainerStarted","Data":"6e17f8ba8b2044c8281508d33e37e302f4f329542d6c904779b67db619b4f566"} Nov 03 20:51:00.832765 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:00.832235 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" event={"ID":"6def77c8-4ce4-4bc5-af6c-599dfe585a00","Type":"ContainerStarted","Data":"44734d3e01779db23c7aa976bf0a035c2a27bb91b7619111b85547f47362b257"} Nov 03 20:51:00.855521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:00.855465 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" podStartSLOduration=1.855451903 podStartE2EDuration="1.855451903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:00.851433635 +0000 UTC m=+2970.381779311" watchObservedRunningTime="2026-11-03 20:51:00.855451903 +0000 UTC m=+2970.385797574" Nov 03 20:51:01.844125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:01.844091 5815 generic.go:356] "Generic (PLEG): container finished" podID="6def77c8-4ce4-4bc5-af6c-599dfe585a00" containerID="6e17f8ba8b2044c8281508d33e37e302f4f329542d6c904779b67db619b4f566" exitCode=0 Nov 03 20:51:01.844660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:01.844176 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" event={"ID":"6def77c8-4ce4-4bc5-af6c-599dfe585a00","Type":"ContainerDied","Data":"6e17f8ba8b2044c8281508d33e37e302f4f329542d6c904779b67db619b4f566"} Nov 03 20:51:03.121985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.121895 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerName="payload-processing" containerID="cri-o://225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966" gracePeriod=25 Nov 03 20:51:03.132983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.132797 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerName="payload-pre-processing" containerID="cri-o://a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f" gracePeriod=25 Nov 03 20:51:03.165542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.165487 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:51:03.255005 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.254972 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca\") pod \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " Nov 03 20:51:03.255229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.255209 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca\") pod \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " Nov 03 20:51:03.255392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.255369 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca" pod "6def77c8-4ce4-4bc5-af6c-599dfe585a00" (UID: "6def77c8-4ce4-4bc5-af6c-599dfe585a00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:03.255449 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.255400 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t\" (UniqueName: \"kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t\") pod \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\" (UID: \"6def77c8-4ce4-4bc5-af6c-599dfe585a00\") " Nov 03 20:51:03.256240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.256213 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:03.257623 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.257599 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca" pod "6def77c8-4ce4-4bc5-af6c-599dfe585a00" (UID: "6def77c8-4ce4-4bc5-af6c-599dfe585a00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:03.257804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.257783 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t" pod "6def77c8-4ce4-4bc5-af6c-599dfe585a00" (UID: "6def77c8-4ce4-4bc5-af6c-599dfe585a00"). InnerVolumeSpecName "kube-api-access-8rz7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:03.358331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.358301 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def77c8-4ce4-4bc5-af6c-599dfe585a00-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:03.358331 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.358330 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rz7t\" (UniqueName: \"kubernetes.io/projected/6def77c8-4ce4-4bc5-af6c-599dfe585a00-kube-api-access-8rz7t\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:03.869134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.869101 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" Nov 03 20:51:03.869134 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.869128 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc" event={"ID":"6def77c8-4ce4-4bc5-af6c-599dfe585a00","Type":"ContainerDied","Data":"44734d3e01779db23c7aa976bf0a035c2a27bb91b7619111b85547f47362b257"} Nov 03 20:51:03.869395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:03.869180 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44734d3e01779db23c7aa976bf0a035c2a27bb91b7619111b85547f47362b257" Nov 03 20:51:04.859614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:04.859574 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:04.860057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:04.859623 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:07.377847 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.377803 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" containerID="cri-o://5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c" gracePeriod=45 Nov 03 20:51:07.766316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.766287 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:51:07.855501 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855465 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache\") pod \"f209fe5c-d690-4126-86f4-d599428d512f\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " Nov 03 20:51:07.855693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855567 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir\") pod \"f209fe5c-d690-4126-86f4-d599428d512f\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " Nov 03 20:51:07.855742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855708 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs\") pod \"f209fe5c-d690-4126-86f4-d599428d512f\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " Nov 03 20:51:07.855742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855724 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache" pod "f209fe5c-d690-4126-86f4-d599428d512f" (UID: "f209fe5c-d690-4126-86f4-d599428d512f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:07.855812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855752 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home\") pod \"f209fe5c-d690-4126-86f4-d599428d512f\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " Nov 03 20:51:07.855812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855794 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir" pod "f209fe5c-d690-4126-86f4-d599428d512f" (UID: "f209fe5c-d690-4126-86f4-d599428d512f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:07.855889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.855861 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt\" (UniqueName: \"kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt\") pod \"f209fe5c-d690-4126-86f4-d599428d512f\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " Nov 03 20:51:07.856025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.856004 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home" pod "f209fe5c-d690-4126-86f4-d599428d512f" (UID: "f209fe5c-d690-4126-86f4-d599428d512f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:07.856151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.856130 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm\") pod \"f209fe5c-d690-4126-86f4-d599428d512f\" (UID: \"f209fe5c-d690-4126-86f4-d599428d512f\") " Nov 03 20:51:07.857053 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.857024 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.857271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.857057 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.857271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.857072 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.858296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.858244 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs" pod "f209fe5c-d690-4126-86f4-d599428d512f" (UID: "f209fe5c-d690-4126-86f4-d599428d512f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:07.858433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.858407 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt" pod "f209fe5c-d690-4126-86f4-d599428d512f" (UID: "f209fe5c-d690-4126-86f4-d599428d512f"). InnerVolumeSpecName "kube-api-access-7p7jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:07.858433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.858420 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm" pod "f209fe5c-d690-4126-86f4-d599428d512f" (UID: "f209fe5c-d690-4126-86f4-d599428d512f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:07.933229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.933190 5815 generic.go:356] "Generic (PLEG): container finished" podID="f209fe5c-d690-4126-86f4-d599428d512f" containerID="5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c" exitCode=0 Nov 03 20:51:07.933409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.933241 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" Nov 03 20:51:07.933409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.933280 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" event={"ID":"f209fe5c-d690-4126-86f4-d599428d512f","Type":"ContainerDied","Data":"5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c"} Nov 03 20:51:07.933409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.933322 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr" event={"ID":"f209fe5c-d690-4126-86f4-d599428d512f","Type":"ContainerDied","Data":"580d19f8d374fb59c980db14456bd28c749498d40f2a36c709a36037a9839f4e"} Nov 03 20:51:07.933409 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.933338 5815 scope.go:122] "RemoveContainer" containerID="5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c" Nov 03 20:51:07.946006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.945988 5815 scope.go:122] "RemoveContainer" containerID="5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c" Nov 03 20:51:07.946306 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:07.946285 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c\": container with ID starting with 5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c not found: ID does not exist" containerID="5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c" Nov 03 20:51:07.946354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.946314 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c"} err="failed to get container status \"5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c\": rpc error: code = NotFound desc = could not find container \"5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c\": container with ID starting with 5946e43e24eef1b0121bad3fffebb248912c584713c813f65d9dea68089a947c not found: ID does not exist" Nov 03 20:51:07.958829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.958804 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p7jt\" (UniqueName: \"kubernetes.io/projected/f209fe5c-d690-4126-86f4-d599428d512f-kube-api-access-7p7jt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.958829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.958829 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f209fe5c-d690-4126-86f4-d599428d512f-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.958829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.958840 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f209fe5c-d690-4126-86f4-d599428d512f-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:07.960122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.960103 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr"] Nov 03 20:51:07.965186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:07.965165 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c94fdf/test-model-c94fdf-kserve-54876c4795-gsmnr"] Nov 03 20:51:08.596484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:08.596403 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f209fe5c-d690-4126-86f4-d599428d512f" path="/var/lib/kubelet/pods/f209fe5c-d690-4126-86f4-d599428d512f/volumes" Nov 03 20:51:09.810849 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.810811 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr"] Nov 03 20:51:09.812174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.812152 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" Nov 03 20:51:09.812247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.812177 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" Nov 03 20:51:09.812247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.812197 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6def77c8-4ce4-4bc5-af6c-599dfe585a00" containerName="revoke-keys" Nov 03 20:51:09.812247 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.812205 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6def77c8-4ce4-4bc5-af6c-599dfe585a00" containerName="revoke-keys" Nov 03 20:51:09.812416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.812403 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f209fe5c-d690-4126-86f4-d599428d512f" containerName="main" Nov 03 20:51:09.812467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.812422 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6def77c8-4ce4-4bc5-af6c-599dfe585a00" containerName="revoke-keys" Nov 03 20:51:09.818066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.818030 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:09.822437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.822407 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr"] Nov 03 20:51:09.823431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.823408 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:09.823431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.823412 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:51:09.912051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.912018 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2s2\" (UniqueName: \"kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:09.912326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.912291 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:09.912555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:09.912501 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.014385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.014352 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.014607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.014477 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2s2\" (UniqueName: \"kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.014689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.014649 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.015118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.015089 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.017252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.017232 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.026388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.026364 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2s2\" (UniqueName: \"kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2\") pod \"maas-api-revoke-sub-9401ec0a-wx5nr\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.140425 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.140346 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:10.430743 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:51:10.430714 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c4cb04_d9fa_42c4_b539_9a97cab3c572.slice/crio-470f8f0baf870befd2ce2542387337bad7208a7538ca7189d780b8b6b4a5006f WatchSource:0}: Error finding container 470f8f0baf870befd2ce2542387337bad7208a7538ca7189d780b8b6b4a5006f: Status 404 returned error can't find the container with id 470f8f0baf870befd2ce2542387337bad7208a7538ca7189d780b8b6b4a5006f Nov 03 20:51:10.431301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:10.431278 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr"] Nov 03 20:51:11.068585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:11.061892 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" event={"ID":"d7c4cb04-d9fa-42c4-b539-9a97cab3c572","Type":"ContainerStarted","Data":"470f8f0baf870befd2ce2542387337bad7208a7538ca7189d780b8b6b4a5006f"} Nov 03 20:51:12.075141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:12.075106 5815 generic.go:356] "Generic (PLEG): container finished" podID="d7c4cb04-d9fa-42c4-b539-9a97cab3c572" containerID="32e4be165d68959d264015bf86e8da3f583869bcfa8d6bfd7286142567ba33b3" exitCode=0 Nov 03 20:51:12.075625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:12.075196 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" event={"ID":"d7c4cb04-d9fa-42c4-b539-9a97cab3c572","Type":"ContainerDied","Data":"32e4be165d68959d264015bf86e8da3f583869bcfa8d6bfd7286142567ba33b3"} Nov 03 20:51:13.289491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.289461 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:13.347095 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.346999 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca\") pod \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " Nov 03 20:51:13.347277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.347128 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2\" (UniqueName: \"kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2\") pod \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " Nov 03 20:51:13.347403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.347387 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca\") pod \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\" (UID: \"d7c4cb04-d9fa-42c4-b539-9a97cab3c572\") " Nov 03 20:51:13.347835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.347803 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca" pod "d7c4cb04-d9fa-42c4-b539-9a97cab3c572" (UID: "d7c4cb04-d9fa-42c4-b539-9a97cab3c572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:13.348213 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.348196 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:13.350077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.350051 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca" pod "d7c4cb04-d9fa-42c4-b539-9a97cab3c572" (UID: "d7c4cb04-d9fa-42c4-b539-9a97cab3c572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:13.350156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.350096 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2" pod "d7c4cb04-d9fa-42c4-b539-9a97cab3c572" (UID: "d7c4cb04-d9fa-42c4-b539-9a97cab3c572"). InnerVolumeSpecName "kube-api-access-8d2s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:13.450760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.450726 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:13.450760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:13.450761 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2s2\" (UniqueName: \"kubernetes.io/projected/d7c4cb04-d9fa-42c4-b539-9a97cab3c572-kube-api-access-8d2s2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:13.854214 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:13.854179 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:51:14.098933 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:14.098904 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" Nov 03 20:51:14.099150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:14.098943 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr" event={"ID":"d7c4cb04-d9fa-42c4-b539-9a97cab3c572","Type":"ContainerDied","Data":"470f8f0baf870befd2ce2542387337bad7208a7538ca7189d780b8b6b4a5006f"} Nov 03 20:51:14.099150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:14.098997 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="470f8f0baf870befd2ce2542387337bad7208a7538ca7189d780b8b6b4a5006f" Nov 03 20:51:14.883349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:14.883312 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:14.900261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:14.900217 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:19.066644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.066613 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk"] Nov 03 20:51:19.067158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.066845 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" containerID="cri-o://200fc2b65faecb1b93ee6bbec27d833baeb8b020041b024e036c4ff216154549" gracePeriod=30 Nov 03 20:51:19.111162 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.111126 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6"] Nov 03 20:51:19.112576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.112477 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d7c4cb04-d9fa-42c4-b539-9a97cab3c572" containerName="revoke-keys" Nov 03 20:51:19.113428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.113383 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c4cb04-d9fa-42c4-b539-9a97cab3c572" containerName="revoke-keys" Nov 03 20:51:19.113788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.113772 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c4cb04-d9fa-42c4-b539-9a97cab3c572" containerName="revoke-keys" Nov 03 20:51:19.129173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.129125 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6"] Nov 03 20:51:19.129369 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.129221 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.137041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.137015 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-745c52-openshift-default-dockercfg-jqchg\"" Nov 03 20:51:19.233065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233026 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233084 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ftc\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233158 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233269 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233202 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233376 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233651 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233711 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233780 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.233817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.233812 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.339995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.339896 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.341716 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.340997 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.356184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.356153 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.356927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.356904 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.357428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.357409 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.358022 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.358002 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.358584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.358564 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ftc\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.358743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.358727 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.358883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.358868 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.368283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.359158 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.370012 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.357355 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.370167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.366101 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.370277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.366321 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.370373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.366547 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.385143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.385117 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.388002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.387978 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ftc\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.394528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.392174 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.430655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.430624 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo\") pod \"e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:19.498193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:19.498160 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:20.144080 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:51:20.144042 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2483d612_5b6d_43e6_936d_8a131aff8ce9.slice/crio-609f77f79554258e54f26d576d49387e1cee2b7608a82a0bc552cbe9e1afe2df WatchSource:0}: Error finding container 609f77f79554258e54f26d576d49387e1cee2b7608a82a0bc552cbe9e1afe2df: Status 404 returned error can't find the container with id 609f77f79554258e54f26d576d49387e1cee2b7608a82a0bc552cbe9e1afe2df Nov 03 20:51:20.146264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.146234 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6"] Nov 03 20:51:20.147474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.147432 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:51:20.147592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.147545 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:51:20.147670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.147594 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:51:20.428632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.422976 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b"] Nov 03 20:51:20.558581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.554012 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" event={"ID":"2483d612-5b6d-43e6-936d-8a131aff8ce9","Type":"ContainerStarted","Data":"fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13"} Nov 03 20:51:20.558581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.554089 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" event={"ID":"2483d612-5b6d-43e6-936d-8a131aff8ce9","Type":"ContainerStarted","Data":"609f77f79554258e54f26d576d49387e1cee2b7608a82a0bc552cbe9e1afe2df"} Nov 03 20:51:20.594905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.592154 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podStartSLOduration=1.592135794 podStartE2EDuration="1.592135794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:20.581929567 +0000 UTC m=+2990.112275239" watchObservedRunningTime="2026-11-03 20:51:20.592135794 +0000 UTC m=+2990.122481467" Nov 03 20:51:20.900565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.890569 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55"] Nov 03 20:51:20.984533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.973623 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:20.984533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.983286 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:20.984533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.983549 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:51:20.984841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:20.984784 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55"] Nov 03 20:51:21.102202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.083650 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb6n\" (UniqueName: \"kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.102202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.083780 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.102202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.083938 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.185474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.185385 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.186385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.186356 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb6n\" (UniqueName: \"kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.186496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.186456 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.187856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.187829 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.194038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.194008 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.202197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.202169 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb6n\" (UniqueName: \"kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n\") pod \"maas-api-revoke-sub-774f6799-l7c55\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.413550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.413476 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:21.510453 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.509787 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:21.526252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.525985 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:21.581917 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.581890 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:21.583451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.583429 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:51:21.730666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:21.715422 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55"] Nov 03 20:51:22.592846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:22.592807 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" event={"ID":"453d6f67-f0da-4e5a-aeda-9511329c9ae0","Type":"ContainerStarted","Data":"d0fd1d7f0d7d0894d14f9588a092d44cdc39d2186d86740e2a637e3634a828df"} Nov 03 20:51:22.593343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:22.592871 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" event={"ID":"453d6f67-f0da-4e5a-aeda-9511329c9ae0","Type":"ContainerStarted","Data":"d1465eefb68a3dcea923a546170ad942087f77472de8e9a745e83a3a6aabd51b"} Nov 03 20:51:22.620427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:22.620372 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" podStartSLOduration=2.620353155 podStartE2EDuration="2.620353155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:22.612752681 +0000 UTC m=+2992.143098354" watchObservedRunningTime="2026-11-03 20:51:22.620353155 +0000 UTC m=+2992.150698829" Nov 03 20:51:23.534431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.534402 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:51:23.548621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.548597 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:51:23.550938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.550918 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume\") pod \"cbcf54e9-a39e-4d5d-b081-0af49229d668\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " Nov 03 20:51:23.551043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.550993 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5\" (UniqueName: \"kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5\") pod \"cbcf54e9-a39e-4d5d-b081-0af49229d668\" (UID: \"cbcf54e9-a39e-4d5d-b081-0af49229d668\") " Nov 03 20:51:23.551580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.551557 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume" pod "cbcf54e9-a39e-4d5d-b081-0af49229d668" (UID: "cbcf54e9-a39e-4d5d-b081-0af49229d668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:23.551930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.551909 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcf54e9-a39e-4d5d-b081-0af49229d668-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.554026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.553988 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5" pod "cbcf54e9-a39e-4d5d-b081-0af49229d668" (UID: "cbcf54e9-a39e-4d5d-b081-0af49229d668"). InnerVolumeSpecName "kube-api-access-twnz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:23.635172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.635090 5815 generic.go:356] "Generic (PLEG): container finished" podID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerID="a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f" exitCode=0 Nov 03 20:51:23.635855 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.635827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" event={"ID":"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6","Type":"ContainerDied","Data":"a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f"} Nov 03 20:51:23.636042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.636023 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" event={"ID":"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6","Type":"ContainerDied","Data":"6177cfb502763535cae6e46cabe89742715be38055a2d1699e9f5694fdbf7966"} Nov 03 20:51:23.636548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.636157 5815 scope.go:122] "RemoveContainer" containerID="a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f" Nov 03 20:51:23.636548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.636361 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj" Nov 03 20:51:23.646419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.645482 5815 generic.go:356] "Generic (PLEG): container finished" podID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerID="225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966" exitCode=0 Nov 03 20:51:23.646419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.645851 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" event={"ID":"cbcf54e9-a39e-4d5d-b081-0af49229d668","Type":"ContainerDied","Data":"225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966"} Nov 03 20:51:23.646419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.645900 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" Nov 03 20:51:23.646419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.645908 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm" event={"ID":"cbcf54e9-a39e-4d5d-b081-0af49229d668","Type":"ContainerDied","Data":"010e597a0d79360ae0fb4c5d43d7ddb817f5de36d80664697bb811ebe16e71d1"} Nov 03 20:51:23.651275 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.651251 5815 generic.go:356] "Generic (PLEG): container finished" podID="453d6f67-f0da-4e5a-aeda-9511329c9ae0" containerID="d0fd1d7f0d7d0894d14f9588a092d44cdc39d2186d86740e2a637e3634a828df" exitCode=0 Nov 03 20:51:23.651491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.651470 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" event={"ID":"453d6f67-f0da-4e5a-aeda-9511329c9ae0","Type":"ContainerDied","Data":"d0fd1d7f0d7d0894d14f9588a092d44cdc39d2186d86740e2a637e3634a828df"} Nov 03 20:51:23.652955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.652934 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx\" (UniqueName: \"kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx\") pod \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " Nov 03 20:51:23.653225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.653206 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume\") pod \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\" (UID: \"b738ce5e-b7ca-45ff-a720-1ff918fa2ee6\") " Nov 03 20:51:23.654469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.654275 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twnz5\" (UniqueName: \"kubernetes.io/projected/cbcf54e9-a39e-4d5d-b081-0af49229d668-kube-api-access-twnz5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.655032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.655013 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume" pod "b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" (UID: "b738ce5e-b7ca-45ff-a720-1ff918fa2ee6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:23.662727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.662698 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx" pod "b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" (UID: "b738ce5e-b7ca-45ff-a720-1ff918fa2ee6"). InnerVolumeSpecName "kube-api-access-bwzpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:23.665754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.664547 5815 scope.go:122] "RemoveContainer" containerID="a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f" Nov 03 20:51:23.665754 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:23.664813 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f\": container with ID starting with a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f not found: ID does not exist" containerID="a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f" Nov 03 20:51:23.665754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.664842 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f"} err="failed to get container status \"a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f\": rpc error: code = NotFound desc = could not find container \"a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f\": container with ID starting with a7d07abf33ab9e1334dfd2608d37c27395e4212bed850e5aae2ff4f7268f456f not found: ID does not exist" Nov 03 20:51:23.665754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.664858 5815 scope.go:122] "RemoveContainer" containerID="225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966" Nov 03 20:51:23.682799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.682717 5815 scope.go:122] "RemoveContainer" containerID="225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966" Nov 03 20:51:23.683678 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:23.683578 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966\": container with ID starting with 225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966 not found: ID does not exist" containerID="225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966" Nov 03 20:51:23.683678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.683616 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966"} err="failed to get container status \"225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966\": rpc error: code = NotFound desc = could not find container \"225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966\": container with ID starting with 225a2ec88069a49e6f98711cd6bd002de34fdfae604aeea76ccd355865159966 not found: ID does not exist" Nov 03 20:51:23.700201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.700172 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm"] Nov 03 20:51:23.708795 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.708772 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d4fe2-59fb4cccd4-zp7tm"] Nov 03 20:51:23.804568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.804539 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwzpx\" (UniqueName: \"kubernetes.io/projected/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-kube-api-access-bwzpx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.804734 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.804571 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.986625 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.986587 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj"] Nov 03 20:51:23.992193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:23.992158 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d4fe2-687c688c79-g64pj"] Nov 03 20:51:24.628314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.628277 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" path="/var/lib/kubelet/pods/b738ce5e-b7ca-45ff-a720-1ff918fa2ee6/volumes" Nov 03 20:51:24.629359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.629335 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" path="/var/lib/kubelet/pods/cbcf54e9-a39e-4d5d-b081-0af49229d668/volumes" Nov 03 20:51:24.680166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.679779 5815 generic.go:356] "Generic (PLEG): container finished" podID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerID="200fc2b65faecb1b93ee6bbec27d833baeb8b020041b024e036c4ff216154549" exitCode=0 Nov 03 20:51:24.680166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.679880 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" event={"ID":"18812c7c-ac28-4334-a508-d73669a0cf0a","Type":"ContainerDied","Data":"200fc2b65faecb1b93ee6bbec27d833baeb8b020041b024e036c4ff216154549"} Nov 03 20:51:24.680166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.679934 5815 scope.go:122] "RemoveContainer" containerID="6e44bb7679d0b8ba9edc7294f5a60371bedc1fd2c8144cdade032ba28ed091b8" Nov 03 20:51:24.863025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.862997 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:51:24.987110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987079 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.987479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987447 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.987648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987482 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:24.987648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987589 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.987792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987697 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.987792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987734 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.987792 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987774 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.987972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.987828 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.988038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.988025 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.988100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.988058 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs\") pod \"18812c7c-ac28-4334-a508-d73669a0cf0a\" (UID: \"18812c7c-ac28-4334-a508-d73669a0cf0a\") " Nov 03 20:51:24.988925 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.988898 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18812c7c-ac28-4334-a508-d73669a0cf0a-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:24.989177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.989136 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:24.989277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.989182 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:24.989277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.989245 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:24.989594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.989570 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:24.993087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.993065 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "kube-api-access-vkhrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:24.993341 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.993314 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:24.993672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:24.993634 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:25.016203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.016164 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo" pod "18812c7c-ac28-4334-a508-d73669a0cf0a" (UID: "18812c7c-ac28-4334-a508-d73669a0cf0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:51:25.074723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.074691 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:25.091157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091125 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091156 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091166 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091175 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091184 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkhrq\" (UniqueName: \"kubernetes.io/projected/18812c7c-ac28-4334-a508-d73669a0cf0a-kube-api-access-vkhrq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091192 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091201 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18812c7c-ac28-4334-a508-d73669a0cf0a-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.091373 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.091210 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18812c7c-ac28-4334-a508-d73669a0cf0a-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.192757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.192724 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca\") pod \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " Nov 03 20:51:25.192935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.192907 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca\") pod \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " Nov 03 20:51:25.193073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.193058 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n\" (UniqueName: \"kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n\") pod \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\" (UID: \"453d6f67-f0da-4e5a-aeda-9511329c9ae0\") " Nov 03 20:51:25.193135 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.193111 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca" pod "453d6f67-f0da-4e5a-aeda-9511329c9ae0" (UID: "453d6f67-f0da-4e5a-aeda-9511329c9ae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:25.193997 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.193971 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.195311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.195285 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca" pod "453d6f67-f0da-4e5a-aeda-9511329c9ae0" (UID: "453d6f67-f0da-4e5a-aeda-9511329c9ae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:25.195433 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.195408 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n" pod "453d6f67-f0da-4e5a-aeda-9511329c9ae0" (UID: "453d6f67-f0da-4e5a-aeda-9511329c9ae0"). InnerVolumeSpecName "kube-api-access-kwb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:25.295434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.295363 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453d6f67-f0da-4e5a-aeda-9511329c9ae0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.295434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.295396 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb6n\" (UniqueName: \"kubernetes.io/projected/453d6f67-f0da-4e5a-aeda-9511329c9ae0-kube-api-access-kwb6n\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:25.722633 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.722604 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" Nov 03 20:51:25.723055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.722654 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk" event={"ID":"18812c7c-ac28-4334-a508-d73669a0cf0a","Type":"ContainerDied","Data":"d2670d078c6d1c3f356b3b81ad7cbb01087cf8de69e4f8d278c8fb6c16c78d1d"} Nov 03 20:51:25.723055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.722722 5815 scope.go:122] "RemoveContainer" containerID="200fc2b65faecb1b93ee6bbec27d833baeb8b020041b024e036c4ff216154549" Nov 03 20:51:25.724845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.724435 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" event={"ID":"453d6f67-f0da-4e5a-aeda-9511329c9ae0","Type":"ContainerDied","Data":"d1465eefb68a3dcea923a546170ad942087f77472de8e9a745e83a3a6aabd51b"} Nov 03 20:51:25.724845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.724462 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55" Nov 03 20:51:25.724845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.724476 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1465eefb68a3dcea923a546170ad942087f77472de8e9a745e83a3a6aabd51b" Nov 03 20:51:25.762321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.762280 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk"] Nov 03 20:51:25.767598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:25.767575 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-kqghk"] Nov 03 20:51:26.612520 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:26.612476 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" path="/var/lib/kubelet/pods/18812c7c-ac28-4334-a508-d73669a0cf0a/volumes" Nov 03 20:51:27.034722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:27.034685 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg"] Nov 03 20:51:27.040255 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:27.040230 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9ae211-8mtwg"] Nov 03 20:51:27.839533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:27.838889 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m"] Nov 03 20:51:27.839533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:27.839213 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" containerID="cri-o://3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057" gracePeriod=30 Nov 03 20:51:28.592649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:28.592562 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea10618-9026-4a7c-8edc-07f6863a8f89" path="/var/lib/kubelet/pods/aea10618-9026-4a7c-8edc-07f6863a8f89/volumes" Nov 03 20:51:31.455076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:31.455042 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp"] Nov 03 20:51:33.218391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.218364 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:51:33.257635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.257595 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.257862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.257664 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.257862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.257756 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.257862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.257817 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.258049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.257880 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.258049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.257969 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.258049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.258032 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.258209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.258129 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.258209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.258196 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert\") pod \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\" (UID: \"b6a28dc7-b27e-4c27-97ee-38996ee945b5\") " Nov 03 20:51:33.258481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.258412 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.258835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.258716 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.258835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.258801 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259036 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259121 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259372 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259394 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259408 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259424 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.259435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.259438 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.261581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.261552 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "kube-api-access-nclqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:33.261705 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.261690 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.261935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.261908 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:33.265954 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.265918 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo" pod "b6a28dc7-b27e-4c27-97ee-38996ee945b5" (UID: "b6a28dc7-b27e-4c27-97ee-38996ee945b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:51:33.361075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.361046 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.361075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.361073 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.361270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.361084 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nclqw\" (UniqueName: \"kubernetes.io/projected/b6a28dc7-b27e-4c27-97ee-38996ee945b5-kube-api-access-nclqw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.361270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.361095 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a28dc7-b27e-4c27-97ee-38996ee945b5-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.842167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.842126 5815 generic.go:356] "Generic (PLEG): container finished" podID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerID="3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057" exitCode=0 Nov 03 20:51:33.842353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.842216 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" event={"ID":"b6a28dc7-b27e-4c27-97ee-38996ee945b5","Type":"ContainerDied","Data":"3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057"} Nov 03 20:51:33.842353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.842231 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" Nov 03 20:51:33.842353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.842276 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m" event={"ID":"b6a28dc7-b27e-4c27-97ee-38996ee945b5","Type":"ContainerDied","Data":"2260d63b661491beedb7401b6b0c2db3d71821e44a2223d7692b4823d3d6c3df"} Nov 03 20:51:33.842353 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.842300 5815 scope.go:122] "RemoveContainer" containerID="3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057" Nov 03 20:51:33.857665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.857646 5815 scope.go:122] "RemoveContainer" containerID="02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1" Nov 03 20:51:33.870159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.870136 5815 scope.go:122] "RemoveContainer" containerID="3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057" Nov 03 20:51:33.870564 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:33.870491 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057\": container with ID starting with 3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057 not found: ID does not exist" containerID="3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057" Nov 03 20:51:33.870687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.870576 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057"} err="failed to get container status \"3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057\": rpc error: code = NotFound desc = could not find container \"3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057\": container with ID starting with 3e84bc8519f262aac27fcc582135b15595466394a5713abd4f47ac3c13209057 not found: ID does not exist" Nov 03 20:51:33.870687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.870606 5815 scope.go:122] "RemoveContainer" containerID="02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1" Nov 03 20:51:33.870927 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:33.870908 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1\": container with ID starting with 02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1 not found: ID does not exist" containerID="02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1" Nov 03 20:51:33.871045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.870934 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1"} err="failed to get container status \"02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1\": rpc error: code = NotFound desc = could not find container \"02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1\": container with ID starting with 02fe17252dcbd16e370bb9155f80e0b49b59a38db373bee7c63ce050a3bda0d1 not found: ID does not exist" Nov 03 20:51:33.874880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.874855 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m"] Nov 03 20:51:33.880649 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:33.880626 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-xp42m"] Nov 03 20:51:34.599136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:34.596176 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" path="/var/lib/kubelet/pods/b6a28dc7-b27e-4c27-97ee-38996ee945b5/volumes" Nov 03 20:51:35.463555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.463480 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:51:35.469769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.469742 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:51:35.470518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.470491 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:51:35.473042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.473005 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" containerID="cri-o://8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631" gracePeriod=45 Nov 03 20:51:35.477666 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.477643 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:51:35.802144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.802118 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.868254 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache\") pod \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.868323 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home\") pod \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.868378 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm\") pod \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.868583 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir\") pod \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.868648 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs\" (UniqueName: \"kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs\") pod \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.868822 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs\" (UniqueName: \"kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs\") pod \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\" (UID: \"088ab3f6-0447-48d9-ac71-bfaa1b8f2791\") " Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.869904 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache" pod "088ab3f6-0447-48d9-ac71-bfaa1b8f2791" (UID: "088ab3f6-0447-48d9-ac71-bfaa1b8f2791"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.870099 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home" pod "088ab3f6-0447-48d9-ac71-bfaa1b8f2791" (UID: "088ab3f6-0447-48d9-ac71-bfaa1b8f2791"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:35.882174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.871027 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir" pod "088ab3f6-0447-48d9-ac71-bfaa1b8f2791" (UID: "088ab3f6-0447-48d9-ac71-bfaa1b8f2791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:35.897706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.897680 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm" pod "088ab3f6-0447-48d9-ac71-bfaa1b8f2791" (UID: "088ab3f6-0447-48d9-ac71-bfaa1b8f2791"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:35.897858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.897840 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs" pod "088ab3f6-0447-48d9-ac71-bfaa1b8f2791" (UID: "088ab3f6-0447-48d9-ac71-bfaa1b8f2791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:35.898768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.898718 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs" pod "088ab3f6-0447-48d9-ac71-bfaa1b8f2791" (UID: "088ab3f6-0447-48d9-ac71-bfaa1b8f2791"). InnerVolumeSpecName "kube-api-access-28zcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:35.971321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.971290 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.971321 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.971319 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.971566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.971330 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.971566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.971340 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.971566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.971348 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.971566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:35.971356 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zcs\" (UniqueName: \"kubernetes.io/projected/088ab3f6-0447-48d9-ac71-bfaa1b8f2791-kube-api-access-28zcs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:36.172108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.171957 5815 generic.go:356] "Generic (PLEG): container finished" podID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerID="8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631" exitCode=0 Nov 03 20:51:36.172108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.172012 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" Nov 03 20:51:36.172108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.172039 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" event={"ID":"088ab3f6-0447-48d9-ac71-bfaa1b8f2791","Type":"ContainerDied","Data":"8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631"} Nov 03 20:51:36.172108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.172092 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" event={"ID":"088ab3f6-0447-48d9-ac71-bfaa1b8f2791","Type":"ContainerDied","Data":"32f9819284bec3befc5ba2a5d30b2154befb637a1c50c44986dca5b2cfe410b2"} Nov 03 20:51:36.172108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.172112 5815 scope.go:122] "RemoveContainer" containerID="8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631" Nov 03 20:51:36.187697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.187675 5815 scope.go:122] "RemoveContainer" containerID="8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631" Nov 03 20:51:36.188012 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:36.187989 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631\": container with ID starting with 8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631 not found: ID does not exist" containerID="8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631" Nov 03 20:51:36.188091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.188018 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631"} err="failed to get container status \"8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631\": rpc error: code = NotFound desc = could not find container \"8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631\": container with ID starting with 8d5a99dbe4d41ad2552a04c9676b8ec15b64d21703b4f87bb8986e6130432631 not found: ID does not exist" Nov 03 20:51:36.201401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.201375 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b"] Nov 03 20:51:36.222539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.220034 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b"] Nov 03 20:51:36.594603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.594569 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" path="/var/lib/kubelet/pods/088ab3f6-0447-48d9-ac71-bfaa1b8f2791/volumes" Nov 03 20:51:36.683777 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:36.683737 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8c6934/test-model-8c6934-kserve-66cdbd5bdf-wzt5b" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" probeResult="failure" output="Get \"https://10.132.1.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:38.038227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:38.038193 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj"] Nov 03 20:51:38.043167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:38.043140 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6c3d40-7fzwj"] Nov 03 20:51:38.983790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:38.983741 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd5092d-8651-4d95-a794-30c12767a10d" path="/var/lib/kubelet/pods/abd5092d-8651-4d95-a794-30c12767a10d/volumes" Nov 03 20:51:42.046077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:42.045278 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw"] Nov 03 20:51:42.046077 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:42.045563 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" containerID="cri-o://e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640" gracePeriod=30 Nov 03 20:51:43.856381 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:43.856343 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:51:45.241528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:45.241478 5815 patch_prober.go:29] interesting pod/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:51:45.242029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:45.241566 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:51:45.910974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:45.910942 5815 scope.go:122] "RemoveContainer" containerID="91950b99b3ba8463e160947994c84a7e2f88d92928ba23da0eeafd28a25cf26c" Nov 03 20:51:45.923679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:45.923657 5815 scope.go:122] "RemoveContainer" containerID="5f597e836513d6d71961dd59e4021522a6c516284590fd72b923a39eb447eecd" Nov 03 20:51:46.106591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.106552 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q"] Nov 03 20:51:46.106796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.106747 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" containerID="cri-o://5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6" gracePeriod=30 Nov 03 20:51:46.537964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.537920 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" containerID="cri-o://af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0" gracePeriod=45 Nov 03 20:51:46.648710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.648653 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" probeResult="failure" output="Get \"https://10.132.1.117:8000/ready\": dial tcp 10.132.1.117:8000: connect: connection refused" Nov 03 20:51:46.917208 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.917175 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:51:46.959141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959110 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir\") pod \"2d409156-98f0-4fcf-9147-b581d5610d17\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " Nov 03 20:51:46.959323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959196 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home\") pod \"2d409156-98f0-4fcf-9147-b581d5610d17\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " Nov 03 20:51:46.959323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959292 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm\") pod \"2d409156-98f0-4fcf-9147-b581d5610d17\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " Nov 03 20:51:46.959548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959324 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs\") pod \"2d409156-98f0-4fcf-9147-b581d5610d17\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " Nov 03 20:51:46.959548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959375 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx\" (UniqueName: \"kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx\") pod \"2d409156-98f0-4fcf-9147-b581d5610d17\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " Nov 03 20:51:46.959548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959390 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir" pod "2d409156-98f0-4fcf-9147-b581d5610d17" (UID: "2d409156-98f0-4fcf-9147-b581d5610d17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:46.959548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.959427 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache\") pod \"2d409156-98f0-4fcf-9147-b581d5610d17\" (UID: \"2d409156-98f0-4fcf-9147-b581d5610d17\") " Nov 03 20:51:46.960361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.960332 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:46.960483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.960399 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home" pod "2d409156-98f0-4fcf-9147-b581d5610d17" (UID: "2d409156-98f0-4fcf-9147-b581d5610d17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:46.960614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.960588 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache" pod "2d409156-98f0-4fcf-9147-b581d5610d17" (UID: "2d409156-98f0-4fcf-9147-b581d5610d17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:46.972732 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.967531 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx" pod "2d409156-98f0-4fcf-9147-b581d5610d17" (UID: "2d409156-98f0-4fcf-9147-b581d5610d17"). InnerVolumeSpecName "kube-api-access-lmbgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:46.976895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.976874 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm" pod "2d409156-98f0-4fcf-9147-b581d5610d17" (UID: "2d409156-98f0-4fcf-9147-b581d5610d17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:46.983780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:46.983760 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs" pod "2d409156-98f0-4fcf-9147-b581d5610d17" (UID: "2d409156-98f0-4fcf-9147-b581d5610d17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:47.061583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.061472 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.061583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.061521 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.061583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.061534 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d409156-98f0-4fcf-9147-b581d5610d17-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.061583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.061548 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbgx\" (UniqueName: \"kubernetes.io/projected/2d409156-98f0-4fcf-9147-b581d5610d17-kube-api-access-lmbgx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.061583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.061563 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d409156-98f0-4fcf-9147-b581d5610d17-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.460803 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.460774 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:51:47.623521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.623471 5815 generic.go:356] "Generic (PLEG): container finished" podID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerID="e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640" exitCode=0 Nov 03 20:51:47.623927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.623612 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" Nov 03 20:51:47.623927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.623640 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" event={"ID":"0c95d6d4-ccfb-4178-a3cb-72700e6de96d","Type":"ContainerDied","Data":"e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640"} Nov 03 20:51:47.623927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.623701 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw" event={"ID":"0c95d6d4-ccfb-4178-a3cb-72700e6de96d","Type":"ContainerDied","Data":"ee335a35b001b08225640fc54fbc64d252562ccaa5d28b2440cff1bd0d19f680"} Nov 03 20:51:47.623927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.623733 5815 scope.go:122] "RemoveContainer" containerID="e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640" Nov 03 20:51:47.626036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626007 5815 generic.go:356] "Generic (PLEG): container finished" podID="2d409156-98f0-4fcf-9147-b581d5610d17" containerID="af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0" exitCode=0 Nov 03 20:51:47.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626043 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" Nov 03 20:51:47.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626075 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" event={"ID":"2d409156-98f0-4fcf-9147-b581d5610d17","Type":"ContainerDied","Data":"af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0"} Nov 03 20:51:47.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626109 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp" event={"ID":"2d409156-98f0-4fcf-9147-b581d5610d17","Type":"ContainerDied","Data":"39d162e4ca1c95aae3a094afc018bcc6e9df6fac1d3de845f1977f4b43fd96a0"} Nov 03 20:51:47.626166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626125 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.626452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626431 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.626587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626571 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.626671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626636 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.626776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626761 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.626930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626906 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:47.627002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626914 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.627002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.626974 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.627120 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.627093 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.627227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.627204 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs\") pod \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\" (UID: \"0c95d6d4-ccfb-4178-a3cb-72700e6de96d\") " Nov 03 20:51:47.628737 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.628709 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.628977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.628929 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:47.629337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.629311 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:47.629595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.629575 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:47.629953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.629926 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:47.631393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.631370 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:47.632339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.632317 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "kube-api-access-tff72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:47.633490 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.633469 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:47.635646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.635626 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo" pod "0c95d6d4-ccfb-4178-a3cb-72700e6de96d" (UID: "0c95d6d4-ccfb-4178-a3cb-72700e6de96d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:51:47.674382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.674358 5815 scope.go:122] "RemoveContainer" containerID="4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1" Nov 03 20:51:47.694985 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.694953 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp"] Nov 03 20:51:47.695327 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.695310 5815 scope.go:122] "RemoveContainer" containerID="e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640" Nov 03 20:51:47.695700 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:47.695677 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640\": container with ID starting with e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640 not found: ID does not exist" containerID="e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640" Nov 03 20:51:47.695797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.695709 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640"} err="failed to get container status \"e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640\": rpc error: code = NotFound desc = could not find container \"e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640\": container with ID starting with e5781293119fec7b71e621d69ecf1090fae5c4d4187075a994b14b8561af7640 not found: ID does not exist" Nov 03 20:51:47.695797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.695727 5815 scope.go:122] "RemoveContainer" containerID="4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1" Nov 03 20:51:47.696026 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:47.696005 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1\": container with ID starting with 4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1 not found: ID does not exist" containerID="4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1" Nov 03 20:51:47.696099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.696034 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1"} err="failed to get container status \"4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1\": rpc error: code = NotFound desc = could not find container \"4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1\": container with ID starting with 4989b5332046f78f3c0bf3406b0c66ac21b70d0a8361e311eaf78c91540d7da1 not found: ID does not exist" Nov 03 20:51:47.696099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.696051 5815 scope.go:122] "RemoveContainer" containerID="af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0" Nov 03 20:51:47.701708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.701654 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f5afd7/test-model-f5afd7-kserve-76fdd9f757-wd9hp"] Nov 03 20:51:47.707500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.707473 5815 scope.go:122] "RemoveContainer" containerID="af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0" Nov 03 20:51:47.707871 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:47.707827 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0\": container with ID starting with af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0 not found: ID does not exist" containerID="af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0" Nov 03 20:51:47.707944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.707876 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0"} err="failed to get container status \"af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0\": rpc error: code = NotFound desc = could not find container \"af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0\": container with ID starting with af7d598efa9f544e1d4f23dd493af57537fcb2ea4fd1f32b0be161104c1018d0 not found: ID does not exist" Nov 03 20:51:47.730482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730457 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730483 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730496 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730529 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730542 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730553 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730564 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tff72\" (UniqueName: \"kubernetes.io/projected/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-kube-api-access-tff72\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.730674 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:47.730575 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c95d6d4-ccfb-4178-a3cb-72700e6de96d-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:48.096094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:48.096063 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw"] Nov 03 20:51:48.125400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:48.125368 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c94fdf-openshift-default-74b9bc76b9-rggnw"] Nov 03 20:51:48.696533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:48.696467 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" path="/var/lib/kubelet/pods/0c95d6d4-ccfb-4178-a3cb-72700e6de96d/volumes" Nov 03 20:51:48.698227 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:48.698186 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" path="/var/lib/kubelet/pods/2d409156-98f0-4fcf-9147-b581d5610d17/volumes" Nov 03 20:51:51.498252 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.498219 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:51:51.589770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.589734 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.589820 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.589901 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590002 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.589953 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.590067 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.590147 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.590219 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590312 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.590298 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.590467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.590392 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs\") pod \"bbd144df-a006-4b38-8d0e-dc3295210007\" (UID: \"bbd144df-a006-4b38-8d0e-dc3295210007\") " Nov 03 20:51:51.591862 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.591834 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:51.592032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.592008 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:51.592418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.592394 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:51.594090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.594059 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:51.606635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.594713 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:51.606635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.595112 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:51.606635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.596812 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:51:51.606635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.599697 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "kube-api-access-d8hv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:51.609152 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.609129 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy" pod "bbd144df-a006-4b38-8d0e-dc3295210007" (UID: "bbd144df-a006-4b38-8d0e-dc3295210007"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:51.693399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693329 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693363 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8hv8\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-kube-api-access-d8hv8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693382 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693395 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd144df-a006-4b38-8d0e-dc3295210007-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693409 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd144df-a006-4b38-8d0e-dc3295210007-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693424 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693437 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd144df-a006-4b38-8d0e-dc3295210007-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693458 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.693677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.693472 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd144df-a006-4b38-8d0e-dc3295210007-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.783027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.782994 5815 generic.go:356] "Generic (PLEG): container finished" podID="bbd144df-a006-4b38-8d0e-dc3295210007" containerID="5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6" exitCode=0 Nov 03 20:51:51.783181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.783055 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" Nov 03 20:51:51.783181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.783073 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" event={"ID":"bbd144df-a006-4b38-8d0e-dc3295210007","Type":"ContainerDied","Data":"5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6"} Nov 03 20:51:51.783181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.783133 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q" event={"ID":"bbd144df-a006-4b38-8d0e-dc3295210007","Type":"ContainerDied","Data":"5b8a83efd42d69edc2298349ffd982f77c2fafcb99034801b462e32befc538b1"} Nov 03 20:51:51.783181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.783160 5815 scope.go:122] "RemoveContainer" containerID="5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6" Nov 03 20:51:51.795912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.795893 5815 scope.go:122] "RemoveContainer" containerID="b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7" Nov 03 20:51:51.808367 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.807997 5815 scope.go:122] "RemoveContainer" containerID="5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6" Nov 03 20:51:51.808730 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:51.808707 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6\": container with ID starting with 5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6 not found: ID does not exist" containerID="5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6" Nov 03 20:51:51.808828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.808749 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6"} err="failed to get container status \"5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6\": rpc error: code = NotFound desc = could not find container \"5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6\": container with ID starting with 5306b6a40b68325826bebc3c93a4f697f2d27fe0dfa48712aee427a6c6c925d6 not found: ID does not exist" Nov 03 20:51:51.808828 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.808767 5815 scope.go:122] "RemoveContainer" containerID="b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7" Nov 03 20:51:51.809081 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:51:51.809062 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7\": container with ID starting with b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7 not found: ID does not exist" containerID="b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7" Nov 03 20:51:51.809129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.809098 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7"} err="failed to get container status \"b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7\": rpc error: code = NotFound desc = could not find container \"b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7\": container with ID starting with b471bb7b573b220fbe87c2e4ecd824b5ada587443573c759b6828431c048e8e7 not found: ID does not exist" Nov 03 20:51:51.816140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.816117 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q"] Nov 03 20:51:51.820877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:51.820855 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-dvv8q"] Nov 03 20:51:52.592978 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:51:52.592940 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" path="/var/lib/kubelet/pods/bbd144df-a006-4b38-8d0e-dc3295210007/volumes" Nov 03 20:52:06.285462 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:06.285418 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:52:13.895535 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:52:13.891036 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:52:14.042768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.042731 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h"] Nov 03 20:52:14.050466 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.050429 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221e46b0-cww4h"] Nov 03 20:52:14.516530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.515945 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99"] Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517275 5815 cpu_manager.go:414] "containerMap: removing container" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517306 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517357 5815 cpu_manager.go:414] "containerMap: removing container" podUID="453d6f67-f0da-4e5a-aeda-9511329c9ae0" containerName="revoke-keys" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517366 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453d6f67-f0da-4e5a-aeda-9511329c9ae0" containerName="revoke-keys" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517374 5815 cpu_manager.go:414] "containerMap: removing container" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517382 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517393 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517401 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517415 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517422 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517441 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517457 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517476 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517484 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517499 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517524 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517546 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerName="payload-pre-processing" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517555 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerName="payload-pre-processing" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517571 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517578 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517613 5815 cpu_manager.go:414] "containerMap: removing container" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerName="payload-processing" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517620 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerName="payload-processing" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517817 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517834 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcf54e9-a39e-4d5d-b081-0af49229d668" containerName="payload-processing" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517849 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517864 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517884 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="453d6f67-f0da-4e5a-aeda-9511329c9ae0" containerName="revoke-keys" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517896 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="088ab3f6-0447-48d9-ac71-bfaa1b8f2791" containerName="main" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517909 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517919 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d409156-98f0-4fcf-9147-b581d5610d17" containerName="main" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517928 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517940 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517952 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c95d6d4-ccfb-4178-a3cb-72700e6de96d" containerName="istio-proxy" Nov 03 20:52:14.521599 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.517964 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b738ce5e-b7ca-45ff-a720-1ff918fa2ee6" containerName="payload-pre-processing" Nov 03 20:52:14.523632 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.522933 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.529810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.529757 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5ecee8-openshift-default-dockercfg-j74pg\"" Nov 03 20:52:14.549222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.547448 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99"] Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.616988 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617113 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617192 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617270 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617391 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617549 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6xd\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617581 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617628 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.618272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.617689 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.638460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.638433 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd9eef0-c0e5-414d-a227-26b7a0e12c2a" path="/var/lib/kubelet/pods/7fd9eef0-c0e5-414d-a227-26b7a0e12c2a/volumes" Nov 03 20:52:14.721537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.720550 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.721537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.720674 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.721537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.720760 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.721537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.720846 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.721537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.720997 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.728709 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.728872 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6xd\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.728910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.728957 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.729031 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.729483 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.733468 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.741604 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.754129 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.780180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.772276 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.785076 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.785015 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.818598 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.813243 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6xd\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.819268 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.819208 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token\") pod \"e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:14.861835 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:14.861803 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:15.554396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:15.554366 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99"] Nov 03 20:52:15.555469 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:52:15.555441 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633a0867_97aa_42ba_a99c_22ec46dc89b8.slice/crio-3884a0337b673378da68f4c6075b435f4e787c9bdd818cf7c41123e9c2c8cfc1 WatchSource:0}: Error finding container 3884a0337b673378da68f4c6075b435f4e787c9bdd818cf7c41123e9c2c8cfc1: Status 404 returned error can't find the container with id 3884a0337b673378da68f4c6075b435f4e787c9bdd818cf7c41123e9c2c8cfc1 Nov 03 20:52:15.559549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:15.559484 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:15.559647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:15.559595 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:15.559647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:15.559642 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:16.347496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:16.347462 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" event={"ID":"633a0867-97aa-42ba-a99c-22ec46dc89b8","Type":"ContainerStarted","Data":"ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10"} Nov 03 20:52:16.347496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:16.347547 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" event={"ID":"633a0867-97aa-42ba-a99c-22ec46dc89b8","Type":"ContainerStarted","Data":"3884a0337b673378da68f4c6075b435f4e787c9bdd818cf7c41123e9c2c8cfc1"} Nov 03 20:52:16.377387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:16.377329 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podStartSLOduration=2.377309799 podStartE2EDuration="2.377309799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:16.371910747 +0000 UTC m=+3045.902256422" watchObservedRunningTime="2026-11-03 20:52:16.377309799 +0000 UTC m=+3045.907655472" Nov 03 20:52:17.054118 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.054080 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:17.350089 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.350003 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:17.436263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.435671 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:17.436263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.436140 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g"] Nov 03 20:52:17.437864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.437392 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" Nov 03 20:52:17.437864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.437415 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd144df-a006-4b38-8d0e-dc3295210007" containerName="istio-proxy" Nov 03 20:52:17.437864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.437498 5815 cpu_manager.go:414] "containerMap: removing container" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" Nov 03 20:52:17.437864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.437526 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18812c7c-ac28-4334-a508-d73669a0cf0a" containerName="istio-proxy" Nov 03 20:52:17.437864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.437724 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a28dc7-b27e-4c27-97ee-38996ee945b5" containerName="istio-proxy" Nov 03 20:52:17.443809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.443788 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:52:17.444015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.444002 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.447816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.447722 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g"] Nov 03 20:52:17.638416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.638331 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.638719 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.638694 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.638876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.638861 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.639156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.639128 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.639244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.639177 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.639244 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.639229 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.639418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.639397 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.639535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.639499 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.639616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.639552 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnvc\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.741563 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.741600 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.741801 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.741959 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742071 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnvc\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742200 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.741958 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742236 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742285 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.742387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742309 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.743013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.742422 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.743872 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.743846 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.744356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.744315 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.745746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.745488 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.745746 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.745710 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.758569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.756660 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnvc\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.758569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.757075 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-pt86g\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:17.788531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:17.782122 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:18.252839 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:52:18.252792 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600d178b_1ec4_458a_b05c_468ad5d6a1f9.slice/crio-cc87c4f233321f2ad1653b90cdf73a279a1b26aec4276ea4d9f5af642e8b8ae6 WatchSource:0}: Error finding container cc87c4f233321f2ad1653b90cdf73a279a1b26aec4276ea4d9f5af642e8b8ae6: Status 404 returned error can't find the container with id cc87c4f233321f2ad1653b90cdf73a279a1b26aec4276ea4d9f5af642e8b8ae6 Nov 03 20:52:18.254090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.254062 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g"] Nov 03 20:52:18.260738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.259771 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:18.260738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.259855 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:18.260738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.259899 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:18.541582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.540221 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" event={"ID":"600d178b-1ec4-458a-b05c-468ad5d6a1f9","Type":"ContainerStarted","Data":"bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65"} Nov 03 20:52:18.541582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.540286 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" event={"ID":"600d178b-1ec4-458a-b05c-468ad5d6a1f9","Type":"ContainerStarted","Data":"cc87c4f233321f2ad1653b90cdf73a279a1b26aec4276ea4d9f5af642e8b8ae6"} Nov 03 20:52:18.803587 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:18.794610 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:19.794100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:19.791768 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:19.794100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:19.791867 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:20.803346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:20.798059 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:20.803346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:20.798161 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:21.803525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:21.802981 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:21.803525 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:21.803056 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:22.625687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:22.625651 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:22.625885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:22.625722 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:22.790875 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:22.790834 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:22.791174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:22.791147 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:23.791437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:23.790916 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:23.791437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:23.791028 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:24.025880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:24.025804 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podStartSLOduration=7.025784163 podStartE2EDuration="7.025784163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:18.58981393 +0000 UTC m=+3048.120159607" watchObservedRunningTime="2026-11-03 20:52:24.025784163 +0000 UTC m=+3053.556129836" Nov 03 20:52:24.031356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:24.031324 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr"] Nov 03 20:52:24.036156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:24.036134 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff493c9c-kl8fr"] Nov 03 20:52:24.613486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:24.613436 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb233ea4-b8fa-4c8e-b651-436450856cf3" path="/var/lib/kubelet/pods/eb233ea4-b8fa-4c8e-b651-436450856cf3/volumes" Nov 03 20:52:24.802133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:24.802097 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:24.802808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:24.802770 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:25.790132 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:25.790097 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:25.790429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:25.790399 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:26.702871 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.702828 5815 generic.go:356] "Generic (PLEG): container finished" podID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerID="fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13" exitCode=0 Nov 03 20:52:26.703471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.702904 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" event={"ID":"2483d612-5b6d-43e6-936d-8a131aff8ce9","Type":"ContainerDied","Data":"fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13"} Nov 03 20:52:26.704020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.704001 5815 scope.go:122] "RemoveContainer" containerID="fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13" Nov 03 20:52:26.706465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.706433 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:26.706603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.706526 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:26.706603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.706572 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:52:26.790349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.790314 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:26.790624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:26.790374 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:27.719725 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:27.719689 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" event={"ID":"2483d612-5b6d-43e6-936d-8a131aff8ce9","Type":"ContainerStarted","Data":"a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6"} Nov 03 20:52:27.797532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:27.796938 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-pt86g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" start-of-body= Nov 03 20:52:27.797532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:27.797015 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.143:15021/healthz/ready\": dial tcp 10.132.1.143:15021: connect: connection refused" Nov 03 20:52:28.502983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:28.502948 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:52:28.504081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:28.503281 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:28.504295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:28.504272 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 20:52:29.512534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:29.512268 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:29.512534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:29.512376 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 20:52:29.620790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:29.620755 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:29.625616 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:29.625569 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:29.631663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:29.631638 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 20:52:30.501979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:30.501932 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:30.502290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:30.502246 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 20:52:31.500100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:31.500059 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:31.500681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:31.500606 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 20:52:32.501040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:32.500927 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:32.501040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:32.500994 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.140:15021/healthz/ready\": dial tcp 10.132.1.140:15021: connect: connection refused" Nov 03 20:52:33.506842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:33.506809 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:52:33.507372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:33.507348 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:52:33.508185 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:33.508164 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:52:36.061618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:36.061577 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk"] Nov 03 20:52:36.073062 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:36.073017 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10ceea9-v9pkk"] Nov 03 20:52:36.590122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:36.590089 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98b60efa-e35e-4659-b74b-5ebb0b67a6f2" path="/var/lib/kubelet/pods/98b60efa-e35e-4659-b74b-5ebb0b67a6f2/volumes" Nov 03 20:52:43.853474 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:52:43.853376 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:52:46.014895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:46.014847 5815 scope.go:122] "RemoveContainer" containerID="fe21b5b97205e6b1c0dc47400863db6511d9b3dc8cf7491ff9590d510f12973b" Nov 03 20:52:46.027038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:46.027017 5815 scope.go:122] "RemoveContainer" containerID="526c62c86685ac619476709f6f391bfca58b902ed6dd72e3095156cc69be840d" Nov 03 20:52:48.028469 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:48.028435 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh"] Nov 03 20:52:48.033339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:48.033310 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-832a34d5-hp8mh"] Nov 03 20:52:48.592257 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:52:48.592218 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03aec59-a797-46b1-a32b-48df42809845" path="/var/lib/kubelet/pods/c03aec59-a797-46b1-a32b-48df42809845/volumes" Nov 03 20:53:00.287834 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:00.287799 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:53:05.741990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.741954 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv"] Nov 03 20:53:05.809989 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.809955 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv"] Nov 03 20:53:05.810313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.810299 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:05.821299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.821272 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-745c52\"/\"kube-root-ca.crt\"" Nov 03 20:53:05.821672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.821654 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-745c52\"/\"openshift-service-ca.crt\"" Nov 03 20:53:05.822533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.822495 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-745c52\"/\"default-dockercfg-bsl9p\"" Nov 03 20:53:05.822820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.822797 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-745c52\"/\"test-model-745c52-kserve-self-signed-certs\"" Nov 03 20:53:05.980628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:05.980575 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.036592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.013687 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srl2x\" (UniqueName: \"kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.036905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.036874 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.037094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.037078 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.041557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.037416 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.109531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.096674 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.214788 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.214750 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.215217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.215190 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srl2x\" (UniqueName: \"kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.215376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.215358 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.220019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.219993 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.220483 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.220464 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.229115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.229092 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.229484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.229430 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.229747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.229727 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.235439 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.235416 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.238382 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.238360 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.238848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.238827 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.251637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.251443 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srl2x\" (UniqueName: \"kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x\") pod \"test-model-745c52-kserve-696c945f87-kxnbv\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.459680 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.459594 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:06.955866 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:53:06.955453 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5fa09b_73f7_445d_9c96_74a8227fd3b5.slice/crio-ea1531c4ac76f9e4a21e11d56f7644c46eaf21c4215f3a139e098ecbd56e7727 WatchSource:0}: Error finding container ea1531c4ac76f9e4a21e11d56f7644c46eaf21c4215f3a139e098ecbd56e7727: Status 404 returned error can't find the container with id ea1531c4ac76f9e4a21e11d56f7644c46eaf21c4215f3a139e098ecbd56e7727 Nov 03 20:53:06.956281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:06.956242 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv"] Nov 03 20:53:07.023936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.023908 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g"] Nov 03 20:53:07.031884 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.030880 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.034522 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.034354 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g"] Nov 03 20:53:07.035124 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.035037 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:07.035230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.035179 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:53:07.159739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.159579 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxffb\" (UniqueName: \"kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.160210 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.160185 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.160463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.160437 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.262072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.262040 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.262253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.262100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.262429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.262407 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxffb\" (UniqueName: \"kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.263172 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.263145 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.292546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.288920 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.310532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.304808 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxffb\" (UniqueName: \"kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb\") pod \"maas-api-revoke-sub-cc47b212-vmh8g\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.354376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.354340 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" event={"ID":"1e5fa09b-73f7-445d-9c96-74a8227fd3b5","Type":"ContainerStarted","Data":"acfabede5d0a0be91799c606734cfedb9578d6aa3448307aa04538b94bac82a2"} Nov 03 20:53:07.354376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.354379 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" event={"ID":"1e5fa09b-73f7-445d-9c96-74a8227fd3b5","Type":"ContainerStarted","Data":"ea1531c4ac76f9e4a21e11d56f7644c46eaf21c4215f3a139e098ecbd56e7727"} Nov 03 20:53:07.426136 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.424828 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:07.463634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.459879 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:07.947962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.947915 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" podStartSLOduration=2.947896295 podStartE2EDuration="2.947896295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:07.381807006 +0000 UTC m=+3096.912152678" watchObservedRunningTime="2026-11-03 20:53:07.947896295 +0000 UTC m=+3097.478242009" Nov 03 20:53:07.948834 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:53:07.948802 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4de4f0c_19aa_4b2b_af0d_839417c6db09.slice/crio-d43786ea8e0a2a93d1cdab68c501dc9c392072ebb9777a4ead4b47b9e0588e87 WatchSource:0}: Error finding container d43786ea8e0a2a93d1cdab68c501dc9c392072ebb9777a4ead4b47b9e0588e87: Status 404 returned error can't find the container with id d43786ea8e0a2a93d1cdab68c501dc9c392072ebb9777a4ead4b47b9e0588e87 Nov 03 20:53:07.949267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:07.949247 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g"] Nov 03 20:53:08.401634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:08.401592 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" event={"ID":"d4de4f0c-19aa-4b2b-af0d-839417c6db09","Type":"ContainerStarted","Data":"d43786ea8e0a2a93d1cdab68c501dc9c392072ebb9777a4ead4b47b9e0588e87"} Nov 03 20:53:09.430390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:09.430353 5815 generic.go:356] "Generic (PLEG): container finished" podID="d4de4f0c-19aa-4b2b-af0d-839417c6db09" containerID="0d6aef1f211ca0174fe2615920f8eec811f5089881d56907cdb54c93bc24421b" exitCode=0 Nov 03 20:53:09.430895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:09.430443 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" event={"ID":"d4de4f0c-19aa-4b2b-af0d-839417c6db09","Type":"ContainerDied","Data":"0d6aef1f211ca0174fe2615920f8eec811f5089881d56907cdb54c93bc24421b"} Nov 03 20:53:10.685827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.685801 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:10.713023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.712994 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca\") pod \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " Nov 03 20:53:10.713256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.713095 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb\" (UniqueName: \"kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb\") pod \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " Nov 03 20:53:10.713256 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.713201 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca\") pod \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\" (UID: \"d4de4f0c-19aa-4b2b-af0d-839417c6db09\") " Nov 03 20:53:10.714383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.714352 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca" pod "d4de4f0c-19aa-4b2b-af0d-839417c6db09" (UID: "d4de4f0c-19aa-4b2b-af0d-839417c6db09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:10.716808 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.716742 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb" pod "d4de4f0c-19aa-4b2b-af0d-839417c6db09" (UID: "d4de4f0c-19aa-4b2b-af0d-839417c6db09"). InnerVolumeSpecName "kube-api-access-bxffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:10.718452 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.718429 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca" pod "d4de4f0c-19aa-4b2b-af0d-839417c6db09" (UID: "d4de4f0c-19aa-4b2b-af0d-839417c6db09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:10.815356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.815313 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxffb\" (UniqueName: \"kubernetes.io/projected/d4de4f0c-19aa-4b2b-af0d-839417c6db09-kube-api-access-bxffb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:10.815356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.815349 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:10.815356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:10.815363 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4de4f0c-19aa-4b2b-af0d-839417c6db09-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:11.456395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:11.456361 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" Nov 03 20:53:11.456395 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:11.456389 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g" event={"ID":"d4de4f0c-19aa-4b2b-af0d-839417c6db09","Type":"ContainerDied","Data":"d43786ea8e0a2a93d1cdab68c501dc9c392072ebb9777a4ead4b47b9e0588e87"} Nov 03 20:53:11.456664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:11.456436 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43786ea8e0a2a93d1cdab68c501dc9c392072ebb9777a4ead4b47b9e0588e87" Nov 03 20:53:13.480260 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:13.480229 5815 generic.go:356] "Generic (PLEG): container finished" podID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerID="ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10" exitCode=0 Nov 03 20:53:13.480877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:13.480306 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" event={"ID":"633a0867-97aa-42ba-a99c-22ec46dc89b8","Type":"ContainerDied","Data":"ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10"} Nov 03 20:53:13.480877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:13.480810 5815 scope.go:122] "RemoveContainer" containerID="ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10" Nov 03 20:53:13.482915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:13.482881 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:53:13.483032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:13.482955 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:53:13.483032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:13.482988 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 20:53:13.856911 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:53:13.856829 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:53:14.492296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:14.492254 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" event={"ID":"633a0867-97aa-42ba-a99c-22ec46dc89b8","Type":"ContainerStarted","Data":"9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545"} Nov 03 20:53:14.877007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:14.876917 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:53:14.877356 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:14.877335 5815 patch_prober.go:29] interesting pod/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" start-of-body= Nov 03 20:53:14.877544 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:14.877519 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" Nov 03 20:53:15.713523 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.713473 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5"] Nov 03 20:53:15.714559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.714536 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d4de4f0c-19aa-4b2b-af0d-839417c6db09" containerName="revoke-keys" Nov 03 20:53:15.714559 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.714559 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4de4f0c-19aa-4b2b-af0d-839417c6db09" containerName="revoke-keys" Nov 03 20:53:15.714783 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.714767 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4de4f0c-19aa-4b2b-af0d-839417c6db09" containerName="revoke-keys" Nov 03 20:53:15.718892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.718871 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.725427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.725406 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ecee8\"/\"openshift-service-ca.crt\"" Nov 03 20:53:15.725762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.725743 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ecee8\"/\"test-model-5ecee8-kserve-self-signed-certs\"" Nov 03 20:53:15.725929 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.725914 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ecee8\"/\"default-dockercfg-qhwr6\"" Nov 03 20:53:15.726220 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.726204 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ecee8\"/\"kube-root-ca.crt\"" Nov 03 20:53:15.726585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.726565 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5"] Nov 03 20:53:15.842276 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.842243 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.842577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.842550 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.842739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.842723 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.842856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.842842 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.843085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.843070 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6h5\" (UniqueName: \"kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.843253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.843237 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:15.874410 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.874381 5815 patch_prober.go:29] interesting pod/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" start-of-body= Nov 03 20:53:15.874678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:15.874650 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" Nov 03 20:53:16.054712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.054621 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.055751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.055722 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.055951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.055934 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.056082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.056069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.056196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.056183 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.056427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.056410 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6h5\" (UniqueName: \"kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.060530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.055289 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.068538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.064899 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.068538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.065144 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.114069 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.114034 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.115148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.115119 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.145580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.141259 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6h5\" (UniqueName: \"kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5\") pod \"test-model-5ecee8-kserve-699b54d677-7k7q5\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.352583 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.348958 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:16.469779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.460073 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:16.544757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.544286 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:16.560905 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.560881 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:16.909072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.908988 5815 patch_prober.go:29] interesting pod/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" start-of-body= Nov 03 20:53:16.909704 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.909671 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" Nov 03 20:53:16.929007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:16.928973 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5"] Nov 03 20:53:17.228372 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.228334 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb"] Nov 03 20:53:17.228636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.228597 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" containerID="cri-o://1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9" gracePeriod=30 Nov 03 20:53:17.435786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.435711 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:53:17.553561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.553486 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" event={"ID":"77ae31d3-cee4-44d9-a25c-381f9f1a5472","Type":"ContainerStarted","Data":"82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f"} Nov 03 20:53:17.553768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.553574 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" event={"ID":"77ae31d3-cee4-44d9-a25c-381f9f1a5472","Type":"ContainerStarted","Data":"7c46de3afb489d81941cbd9e32951f99a1fcdb8bf794647ad9362340159a7593"} Nov 03 20:53:17.887530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.883548 5815 patch_prober.go:29] interesting pod/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" start-of-body= Nov 03 20:53:17.887530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:17.883602 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" Nov 03 20:53:18.351747 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:18.351713 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:18.877621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:18.877587 5815 patch_prober.go:29] interesting pod/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" start-of-body= Nov 03 20:53:18.877912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:18.877881 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" Nov 03 20:53:19.896381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:19.895933 5815 patch_prober.go:29] interesting pod/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" start-of-body= Nov 03 20:53:19.896381 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:19.895993 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.142:15021/healthz/ready\": dial tcp 10.132.1.142:15021: connect: connection refused" Nov 03 20:53:20.885979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:20.885946 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:53:20.887277 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:20.887256 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:53:20.914990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:20.914947 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" podStartSLOduration=5.914934399 podStartE2EDuration="5.914934399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:17.595770522 +0000 UTC m=+3107.126116196" watchObservedRunningTime="2026-11-03 20:53:20.914934399 +0000 UTC m=+3110.445280071" Nov 03 20:53:22.199263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.199221 5815 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:53:22.199891 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.199291 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:53:22.608307 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.608276 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:53:22.644184 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644153 5815 generic.go:356] "Generic (PLEG): container finished" podID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerID="1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9" exitCode=0 Nov 03 20:53:22.644386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644196 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" Nov 03 20:53:22.644386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644236 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" event={"ID":"b1199af3-96ca-4907-9aff-cd92be8c8427","Type":"ContainerDied","Data":"1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9"} Nov 03 20:53:22.644386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644293 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb" event={"ID":"b1199af3-96ca-4907-9aff-cd92be8c8427","Type":"ContainerDied","Data":"a344140047fc9b80dd4c4dde4347a317c36b9fbb8511e354354aead5409546a6"} Nov 03 20:53:22.644386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644321 5815 scope.go:122] "RemoveContainer" containerID="1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9" Nov 03 20:53:22.644386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644323 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.644727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644425 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.644727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644554 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.644727 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644626 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.644892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644762 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:22.644892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644778 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.644892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644835 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.645055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644928 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.645055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.644983 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.645055 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.645045 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert\") pod \"b1199af3-96ca-4907-9aff-cd92be8c8427\" (UID: \"b1199af3-96ca-4907-9aff-cd92be8c8427\") " Nov 03 20:53:22.645767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.645677 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:22.645767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.645729 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:22.646004 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.645982 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.646081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.646011 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.646081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.646025 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.646178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.646063 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:22.646178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.646086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:22.647488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.647465 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:22.647839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.647816 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:53:22.647927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.647902 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:22.649074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.649054 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff" pod "b1199af3-96ca-4907-9aff-cd92be8c8427" (UID: "b1199af3-96ca-4907-9aff-cd92be8c8427"). InnerVolumeSpecName "kube-api-access-kckff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:22.698435 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.698406 5815 scope.go:122] "RemoveContainer" containerID="1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9" Nov 03 20:53:22.698872 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:53:22.698851 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9\": container with ID starting with 1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9 not found: ID does not exist" containerID="1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9" Nov 03 20:53:22.698986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.698884 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9"} err="failed to get container status \"1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9\": rpc error: code = NotFound desc = could not find container \"1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9\": container with ID starting with 1701e7515b907f244e5eb9660310574e0fc51593ac833fd9bda45127359439c9 not found: ID does not exist" Nov 03 20:53:22.749355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.748480 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.749355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.748535 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.749355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.748549 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.749355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.748563 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckff\" (UniqueName: \"kubernetes.io/projected/b1199af3-96ca-4907-9aff-cd92be8c8427-kube-api-access-kckff\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.749355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.748579 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1199af3-96ca-4907-9aff-cd92be8c8427-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.749355 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.748592 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1199af3-96ca-4907-9aff-cd92be8c8427-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:22.972112 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.972082 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb"] Nov 03 20:53:22.977957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:22.977932 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-9z8mb"] Nov 03 20:53:24.593947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:24.593904 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" path="/var/lib/kubelet/pods/b1199af3-96ca-4907-9aff-cd92be8c8427/volumes" Nov 03 20:53:25.939631 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.939595 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j"] Nov 03 20:53:25.940555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.940481 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" Nov 03 20:53:25.940555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.940521 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" Nov 03 20:53:25.940741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.940722 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1199af3-96ca-4907-9aff-cd92be8c8427" containerName="istio-proxy" Nov 03 20:53:25.946689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.946202 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:25.953091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.952493 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:25.953091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.952556 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:53:25.953091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:25.952871 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j"] Nov 03 20:53:26.003250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.003209 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.003404 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.003261 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.003450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.003397 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txlhb\" (UniqueName: \"kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.105708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.105669 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.105869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.105725 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.105869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.105791 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txlhb\" (UniqueName: \"kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.106427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.106357 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.108548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.108480 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.119165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.119145 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txlhb\" (UniqueName: \"kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb\") pod \"maas-api-revoke-sub-4857b0b9-dwv4j\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.274966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.274920 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:26.351885 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.351858 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:26.368428 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.368403 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:26.593376 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:53:26.593331 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27389859_8742_44a6_b6c9_372928569080.slice/crio-03f21267f2308a4d5de55c705327f980bfe54039adc4f7343399ae5962b2db6b WatchSource:0}: Error finding container 03f21267f2308a4d5de55c705327f980bfe54039adc4f7343399ae5962b2db6b: Status 404 returned error can't find the container with id 03f21267f2308a4d5de55c705327f980bfe54039adc4f7343399ae5962b2db6b Nov 03 20:53:26.595825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.595794 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j"] Nov 03 20:53:26.686065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.686030 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" event={"ID":"27389859-8742-44a6-b6c9-372928569080","Type":"ContainerStarted","Data":"03f21267f2308a4d5de55c705327f980bfe54039adc4f7343399ae5962b2db6b"} Nov 03 20:53:26.697824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:26.697800 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:27.699133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:27.699099 5815 generic.go:356] "Generic (PLEG): container finished" podID="27389859-8742-44a6-b6c9-372928569080" containerID="ce7ce7418ad1438e3bcea36ea7645d6ae94a872f1518516ee726ec1f2cf5307f" exitCode=0 Nov 03 20:53:27.699665 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:27.699176 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" event={"ID":"27389859-8742-44a6-b6c9-372928569080","Type":"ContainerDied","Data":"ce7ce7418ad1438e3bcea36ea7645d6ae94a872f1518516ee726ec1f2cf5307f"} Nov 03 20:53:28.915630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.915607 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:28.957548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.957471 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb\" (UniqueName: \"kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb\") pod \"27389859-8742-44a6-b6c9-372928569080\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " Nov 03 20:53:28.957739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.957592 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca\") pod \"27389859-8742-44a6-b6c9-372928569080\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " Nov 03 20:53:28.957739 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.957669 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca\") pod \"27389859-8742-44a6-b6c9-372928569080\" (UID: \"27389859-8742-44a6-b6c9-372928569080\") " Nov 03 20:53:28.958225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.958196 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca" pod "27389859-8742-44a6-b6c9-372928569080" (UID: "27389859-8742-44a6-b6c9-372928569080"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:28.958692 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.958663 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27389859-8742-44a6-b6c9-372928569080-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:28.960481 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.960447 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb" pod "27389859-8742-44a6-b6c9-372928569080" (UID: "27389859-8742-44a6-b6c9-372928569080"). InnerVolumeSpecName "kube-api-access-txlhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:28.960756 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:28.960732 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca" pod "27389859-8742-44a6-b6c9-372928569080" (UID: "27389859-8742-44a6-b6c9-372928569080"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:29.060622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:29.060591 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txlhb\" (UniqueName: \"kubernetes.io/projected/27389859-8742-44a6-b6c9-372928569080-kube-api-access-txlhb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:29.060622 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:29.060620 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27389859-8742-44a6-b6c9-372928569080-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:29.719881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:29.719838 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" Nov 03 20:53:29.719881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:29.719863 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j" event={"ID":"27389859-8742-44a6-b6c9-372928569080","Type":"ContainerDied","Data":"03f21267f2308a4d5de55c705327f980bfe54039adc4f7343399ae5962b2db6b"} Nov 03 20:53:29.720121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:29.719906 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f21267f2308a4d5de55c705327f980bfe54039adc4f7343399ae5962b2db6b" Nov 03 20:53:36.525181 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.525143 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh"] Nov 03 20:53:36.936860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.936780 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m"] Nov 03 20:53:36.938094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.938071 5815 cpu_manager.go:414] "containerMap: removing container" podUID="27389859-8742-44a6-b6c9-372928569080" containerName="revoke-keys" Nov 03 20:53:36.938191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.938098 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27389859-8742-44a6-b6c9-372928569080" containerName="revoke-keys" Nov 03 20:53:36.938330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.938316 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="27389859-8742-44a6-b6c9-372928569080" containerName="revoke-keys" Nov 03 20:53:36.943104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.943079 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:36.948816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.948778 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:36.948964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.948813 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:53:36.948964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:36.948856 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m"] Nov 03 20:53:37.073621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.073586 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.073981 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.073963 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9z4\" (UniqueName: \"kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.074042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.074024 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.183776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.182194 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9z4\" (UniqueName: \"kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.183776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.182302 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.183776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.182428 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.184844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.184816 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.188927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.188865 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.196718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.196696 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9z4\" (UniqueName: \"kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4\") pod \"maas-api-revoke-sub-7aad716f-zwt7m\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.265094 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.265061 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:37.573292 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:53:37.573259 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21519c85_ce0b_436a_b195_a454392392e9.slice/crio-9aa9a4469e0c816ccebc47bf174ad62ba08d0614449f4fe13af6e56620e6535f WatchSource:0}: Error finding container 9aa9a4469e0c816ccebc47bf174ad62ba08d0614449f4fe13af6e56620e6535f: Status 404 returned error can't find the container with id 9aa9a4469e0c816ccebc47bf174ad62ba08d0614449f4fe13af6e56620e6535f Nov 03 20:53:37.573741 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.573459 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m"] Nov 03 20:53:37.883629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:37.882854 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" event={"ID":"21519c85-ce0b-436a-b195-a454392392e9","Type":"ContainerStarted","Data":"9aa9a4469e0c816ccebc47bf174ad62ba08d0614449f4fe13af6e56620e6535f"} Nov 03 20:53:38.898706 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:38.898665 5815 generic.go:356] "Generic (PLEG): container finished" podID="21519c85-ce0b-436a-b195-a454392392e9" containerID="03e27adf8af20b3a421cd18794f2a9b6afa76278df644b345dd6ca427564f4c2" exitCode=0 Nov 03 20:53:38.899242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:38.898744 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" event={"ID":"21519c85-ce0b-436a-b195-a454392392e9","Type":"ContainerDied","Data":"03e27adf8af20b3a421cd18794f2a9b6afa76278df644b345dd6ca427564f4c2"} Nov 03 20:53:40.179319 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.179254 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:40.183271 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.183249 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca\") pod \"21519c85-ce0b-436a-b195-a454392392e9\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " Nov 03 20:53:40.183391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.183331 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4\" (UniqueName: \"kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4\") pod \"21519c85-ce0b-436a-b195-a454392392e9\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " Nov 03 20:53:40.183785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.183754 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca\") pod \"21519c85-ce0b-436a-b195-a454392392e9\" (UID: \"21519c85-ce0b-436a-b195-a454392392e9\") " Nov 03 20:53:40.184164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.184136 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca" pod "21519c85-ce0b-436a-b195-a454392392e9" (UID: "21519c85-ce0b-436a-b195-a454392392e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:40.184797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.184773 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21519c85-ce0b-436a-b195-a454392392e9-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:40.185672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.185648 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca" pod "21519c85-ce0b-436a-b195-a454392392e9" (UID: "21519c85-ce0b-436a-b195-a454392392e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:40.185952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.185936 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4" pod "21519c85-ce0b-436a-b195-a454392392e9" (UID: "21519c85-ce0b-436a-b195-a454392392e9"). InnerVolumeSpecName "kube-api-access-nx9z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:40.286644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.286608 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21519c85-ce0b-436a-b195-a454392392e9-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:40.286644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.286639 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9z4\" (UniqueName: \"kubernetes.io/projected/21519c85-ce0b-436a-b195-a454392392e9-kube-api-access-nx9z4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:40.921358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.921278 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" Nov 03 20:53:40.921358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.921307 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m" event={"ID":"21519c85-ce0b-436a-b195-a454392392e9","Type":"ContainerDied","Data":"9aa9a4469e0c816ccebc47bf174ad62ba08d0614449f4fe13af6e56620e6535f"} Nov 03 20:53:40.921358 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:40.921356 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aa9a4469e0c816ccebc47bf174ad62ba08d0614449f4fe13af6e56620e6535f" Nov 03 20:53:43.856000 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:53:43.855964 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:53:45.850558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:45.850523 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:53:46.091864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:46.091836 5815 scope.go:122] "RemoveContainer" containerID="0a102317872cb013dbd08480479ffe6ce90e713ff34361daa8acaea3be676c09" Nov 03 20:53:46.108816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:46.108796 5815 scope.go:122] "RemoveContainer" containerID="55702f0864ad41237b8c7283194db224d7d3eda7a9f22f2529e6803c8c5110cd" Nov 03 20:53:46.122361 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:46.122343 5815 scope.go:122] "RemoveContainer" containerID="6c67059eeb0dd78be729ce9230b9dfaa58dce1c55f32d5bdaf91c552096b27a9" Nov 03 20:53:46.134693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:46.134670 5815 scope.go:122] "RemoveContainer" containerID="d6f255425b3194d126b3e1ed6a95329b80bba23938725dcb272ffef91eaac09b" Nov 03 20:53:46.145764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:46.145740 5815 scope.go:122] "RemoveContainer" containerID="733a95c14860d1655ee76fc003a1080a4bab02b251216bf21a67c6786ec9422a" Nov 03 20:53:47.942534 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:47.941632 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh"] Nov 03 20:53:48.552584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.527585 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 20:53:48.554877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.554835 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.144:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:48.748563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.742601 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.132:8000/ready\": context deadline exceeded" Nov 03 20:53:48.754243 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.754024 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:53:48.756548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.756497 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" probeResult="failure" output="Get \"https://10.132.1.146:8000/ready\": context deadline exceeded" Nov 03 20:53:48.810589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.794644 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.131:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:48.845629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.845583 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:48.902896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:48.902841 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.340696 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.340653 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" podUID="c34b18ef-3f1a-4291-849f-8a297b233b93" containerName="main" probeResult="failure" output="Get \"https://10.132.1.134:8000/ready\": context deadline exceeded" Nov 03 20:53:49.530926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.530881 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.531751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.531726 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:53:49.673588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.648088 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.770584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.766712 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" probeResult="failure" output="Get \"https://10.132.1.146:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.792566 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.771427 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:49.792813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.777761 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.793173 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.777792 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.132:8000/ready\": context deadline exceeded" Nov 03 20:53:49.823273 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.784572 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.823916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.823885 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:53:49.825221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.825203 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:53:49.825733 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.803030 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.826154 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.826138 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:53:49.847775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.845970 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:49.877537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:49.872921 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:53:50.097203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.097162 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" podUID="90948292-dd9b-4b36-8608-000229f5c249" containerName="main" probeResult="failure" output="Get \"https://10.132.1.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:50.334049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.333924 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 20:53:50.423660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.423564 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:50.664920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.664610 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.144:8000/ready\": context deadline exceeded" Nov 03 20:53:50.693431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.692464 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 20:53:50.801342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.800571 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:50.820995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.801983 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:53:50.820995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.802040 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:53:50.820995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.802071 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:53:50.820995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.802105 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:53:50.824179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.823259 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.132:8000/ready\": context deadline exceeded" Nov 03 20:53:50.824179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.823942 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:50.827105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.826283 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:50.837535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.835441 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 20:53:50.948698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:50.948605 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:51.042611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.037084 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:53:51.042611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.037204 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:53:51.098634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.098587 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" podUID="90948292-dd9b-4b36-8608-000229f5c249" containerName="main" probeResult="failure" output="Get \"https://10.132.1.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:51.135248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.126200 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:53:51.257478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.257439 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:53:51.614591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.610982 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 20:53:51.658581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.658543 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" containerID="cri-o://d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265" gracePeriod=45 Nov 03 20:53:51.939195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.938342 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:51.997562 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.996937 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.144:8000/ready\": context deadline exceeded" Nov 03 20:53:51.998080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:51.997959 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:52.038572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.022406 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.131:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:52.038572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.023140 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.131:8000/ready\": dial tcp 10.132.1.131:8000: connect: connection refused" Nov 03 20:53:52.192712 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.175302 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:53:52.628455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.628422 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:53:52.640398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640368 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir\") pod \"a0496e47-f1e4-4d10-b865-787fe8338fb7\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " Nov 03 20:53:52.640615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640474 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache\") pod \"a0496e47-f1e4-4d10-b865-787fe8338fb7\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " Nov 03 20:53:52.640615 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640548 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home\") pod \"a0496e47-f1e4-4d10-b865-787fe8338fb7\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " Nov 03 20:53:52.640786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640663 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs\") pod \"a0496e47-f1e4-4d10-b865-787fe8338fb7\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " Nov 03 20:53:52.640786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640697 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc\" (UniqueName: \"kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc\") pod \"a0496e47-f1e4-4d10-b865-787fe8338fb7\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " Nov 03 20:53:52.640786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640765 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm\") pod \"a0496e47-f1e4-4d10-b865-787fe8338fb7\" (UID: \"a0496e47-f1e4-4d10-b865-787fe8338fb7\") " Nov 03 20:53:52.640976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.640793 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir" pod "a0496e47-f1e4-4d10-b865-787fe8338fb7" (UID: "a0496e47-f1e4-4d10-b865-787fe8338fb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:52.642450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.641230 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home" pod "a0496e47-f1e4-4d10-b865-787fe8338fb7" (UID: "a0496e47-f1e4-4d10-b865-787fe8338fb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:52.642450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.641740 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache" pod "a0496e47-f1e4-4d10-b865-787fe8338fb7" (UID: "a0496e47-f1e4-4d10-b865-787fe8338fb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:52.642450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.642223 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:52.642450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.642403 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:52.642450 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.642440 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:52.643796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.643773 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc" pod "a0496e47-f1e4-4d10-b865-787fe8338fb7" (UID: "a0496e47-f1e4-4d10-b865-787fe8338fb7"). InnerVolumeSpecName "kube-api-access-j7mmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:52.643967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.643944 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm" pod "a0496e47-f1e4-4d10-b865-787fe8338fb7" (UID: "a0496e47-f1e4-4d10-b865-787fe8338fb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:52.644023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.643959 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs" pod "a0496e47-f1e4-4d10-b865-787fe8338fb7" (UID: "a0496e47-f1e4-4d10-b865-787fe8338fb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:52.743793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.743714 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0496e47-f1e4-4d10-b865-787fe8338fb7-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:52.743793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.743748 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7mmc\" (UniqueName: \"kubernetes.io/projected/a0496e47-f1e4-4d10-b865-787fe8338fb7-kube-api-access-j7mmc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:52.743793 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:52.743762 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0496e47-f1e4-4d10-b865-787fe8338fb7-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:53.025499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.022324 5815 generic.go:356] "Generic (PLEG): container finished" podID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerID="d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265" exitCode=0 Nov 03 20:53:53.025499 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.023362 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" Nov 03 20:53:53.026723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.026688 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" event={"ID":"a0496e47-f1e4-4d10-b865-787fe8338fb7","Type":"ContainerDied","Data":"d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265"} Nov 03 20:53:53.026865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.026755 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh" event={"ID":"a0496e47-f1e4-4d10-b865-787fe8338fb7","Type":"ContainerDied","Data":"e51704becbe841deba26530ae0252092b37038022ecb43cefa7cf76cc1e4e73f"} Nov 03 20:53:53.026865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.026778 5815 scope.go:122] "RemoveContainer" containerID="d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265" Nov 03 20:53:53.044129 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.044065 5815 scope.go:122] "RemoveContainer" containerID="d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265" Nov 03 20:53:53.044386 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:53:53.044332 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265\": container with ID starting with d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265 not found: ID does not exist" containerID="d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265" Nov 03 20:53:53.044386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.044356 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265"} err="failed to get container status \"d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265\": rpc error: code = NotFound desc = could not find container \"d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265\": container with ID starting with d27209fe8f1465a73c5f6bf573c1d13dd97fcea5974890371041861157cbb265 not found: ID does not exist" Nov 03 20:53:53.063101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.063069 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh"] Nov 03 20:53:53.071370 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:53.071338 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bf7965/test-model-bf7965-kserve-6fb5fbf5d9-4whzh"] Nov 03 20:53:54.599106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:53:54.599072 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" path="/var/lib/kubelet/pods/a0496e47-f1e4-4d10-b865-787fe8338fb7/volumes" Nov 03 20:54:06.571109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:06.571063 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" containerID="cri-o://64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a" gracePeriod=42 Nov 03 20:54:06.728809 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:06.728770 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" probeResult="failure" output="Get \"https://10.132.1.132:8000/ready\": dial tcp 10.132.1.132:8000: connect: connection refused" Nov 03 20:54:06.987812 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:06.987766 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:54:07.078561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.078526 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb\" (UniqueName: \"kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb\") pod \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " Nov 03 20:54:07.078770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.078607 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm\") pod \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " Nov 03 20:54:07.078770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.078665 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs\") pod \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " Nov 03 20:54:07.078922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.078886 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home\") pod \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " Nov 03 20:54:07.078982 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.078949 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache\") pod \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " Nov 03 20:54:07.079057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.079043 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir\") pod \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\" (UID: \"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c\") " Nov 03 20:54:07.079229 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.079201 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home" pod "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" (UID: "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:07.079392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.079365 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache" pod "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" (UID: "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:07.079479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.079464 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir" pod "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" (UID: "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:07.080205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.080182 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:07.080317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.080211 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:07.088582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.088535 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm" pod "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" (UID: "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:07.088865 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.088835 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb" pod "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" (UID: "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c"). InnerVolumeSpecName "kube-api-access-rxsvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:07.091057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.091033 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs" pod "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" (UID: "b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:07.181302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.181230 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxsvb\" (UniqueName: \"kubernetes.io/projected/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-kube-api-access-rxsvb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:07.181302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.181261 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:07.181302 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.181298 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:07.181569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.181311 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:07.325148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.325118 5815 generic.go:356] "Generic (PLEG): container finished" podID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerID="64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a" exitCode=0 Nov 03 20:54:07.325335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.325165 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" Nov 03 20:54:07.325335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.325208 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" event={"ID":"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c","Type":"ContainerDied","Data":"64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a"} Nov 03 20:54:07.325335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.325255 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh" event={"ID":"b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c","Type":"ContainerDied","Data":"b56e9f177413f3bfa90a87a4b632ed903bab8a18137a9593d2d921707707b559"} Nov 03 20:54:07.325335 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.325277 5815 scope.go:122] "RemoveContainer" containerID="64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a" Nov 03 20:54:07.338546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.338500 5815 scope.go:122] "RemoveContainer" containerID="64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a" Nov 03 20:54:07.338902 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:07.338859 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a\": container with ID starting with 64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a not found: ID does not exist" containerID="64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a" Nov 03 20:54:07.339024 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.338903 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a"} err="failed to get container status \"64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a\": rpc error: code = NotFound desc = could not find container \"64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a\": container with ID starting with 64612746217eefaca2fa3ecc44d533253d0f999d81583b1d1db34a72d032d46a not found: ID does not exist" Nov 03 20:54:07.353189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.353164 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh"] Nov 03 20:54:07.357991 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:07.357971 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d7c7fd/test-model-d7c7fd-kserve-7d8d5b6645-c4mrh"] Nov 03 20:54:08.593285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:08.592986 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" path="/var/lib/kubelet/pods/b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c/volumes" Nov 03 20:54:13.854801 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:13.854762 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:54:17.478667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:17.478617 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z"] Nov 03 20:54:17.479614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:17.479560 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" containerID="cri-o://7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307" gracePeriod=30 Nov 03 20:54:22.782819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.782791 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:54:22.923073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923041 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923110 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923146 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923179 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923281 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923325 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923350 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923399 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923344 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:22.923661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923437 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923579 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:22.923661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923618 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token\") pod \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\" (UID: \"6a47b882-1e6c-4aef-8b71-9e1b8629b857\") " Nov 03 20:54:22.923661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923603 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:22.923888 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923797 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:22.923968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.923945 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:22.924762 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.924739 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:22.924857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.924769 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:22.924857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.924786 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:22.924857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.924801 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:22.924857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.924814 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:22.925893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.925868 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:22.925986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.925953 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:22.926099 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.926082 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:54:22.926174 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:22.926157 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8" pod "6a47b882-1e6c-4aef-8b71-9e1b8629b857" (UID: "6a47b882-1e6c-4aef-8b71-9e1b8629b857"). InnerVolumeSpecName "kube-api-access-9tks8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:23.026617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.026490 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tks8\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-kube-api-access-9tks8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:23.026617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.026563 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:23.026617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.026579 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:23.026617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.026590 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a47b882-1e6c-4aef-8b71-9e1b8629b857-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:23.603230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.603157 5815 generic.go:356] "Generic (PLEG): container finished" podID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerID="7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307" exitCode=0 Nov 03 20:54:23.603230 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.603224 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" event={"ID":"6a47b882-1e6c-4aef-8b71-9e1b8629b857","Type":"ContainerDied","Data":"7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307"} Nov 03 20:54:23.603497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.603235 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" Nov 03 20:54:23.603497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.603278 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z" event={"ID":"6a47b882-1e6c-4aef-8b71-9e1b8629b857","Type":"ContainerDied","Data":"65f4ef6347e7e33c6543acf06b8d9e91b7326effc0086f1c5ba8fb04bb508c0c"} Nov 03 20:54:23.603497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.603306 5815 scope.go:122] "RemoveContainer" containerID="7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307" Nov 03 20:54:23.619962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.619899 5815 scope.go:122] "RemoveContainer" containerID="b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9" Nov 03 20:54:23.633090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.633065 5815 scope.go:122] "RemoveContainer" containerID="7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307" Nov 03 20:54:23.633355 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:23.633331 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307\": container with ID starting with 7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307 not found: ID does not exist" containerID="7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307" Nov 03 20:54:23.633408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.633365 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307"} err="failed to get container status \"7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307\": rpc error: code = NotFound desc = could not find container \"7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307\": container with ID starting with 7ac006cc2adccf168fc7c32a62200a814872d1dcd51470e00120b6c46faea307 not found: ID does not exist" Nov 03 20:54:23.633408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.633385 5815 scope.go:122] "RemoveContainer" containerID="b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9" Nov 03 20:54:23.633640 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:23.633622 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9\": container with ID starting with b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9 not found: ID does not exist" containerID="b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9" Nov 03 20:54:23.633698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.633646 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9"} err="failed to get container status \"b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9\": rpc error: code = NotFound desc = could not find container \"b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9\": container with ID starting with b513e20dd6b31377a8c69b0bdb791907b1461dc2714c9d7e92b6846968207db9 not found: ID does not exist" Nov 03 20:54:23.641032 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.641001 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z"] Nov 03 20:54:23.646178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:23.646153 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bf7965-openshift-default-79fb49f54f-96h4z"] Nov 03 20:54:24.590266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:24.589971 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" path="/var/lib/kubelet/pods/6a47b882-1e6c-4aef-8b71-9e1b8629b857/volumes" Nov 03 20:54:25.228790 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:25.228750 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q"] Nov 03 20:54:25.246091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:25.246055 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7"] Nov 03 20:54:26.565303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.565271 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn"] Nov 03 20:54:26.566093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566077 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566095 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566111 5815 cpu_manager.go:414] "containerMap: removing container" podUID="21519c85-ce0b-436a-b195-a454392392e9" containerName="revoke-keys" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566116 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21519c85-ce0b-436a-b195-a454392392e9" containerName="revoke-keys" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566126 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566131 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566146 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" Nov 03 20:54:26.566151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566151 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" Nov 03 20:54:26.566380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566175 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" Nov 03 20:54:26.566380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566180 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" Nov 03 20:54:26.566380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566325 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97a9c7a-41e7-4d61-b0c0-0ad5dfbb0e6c" containerName="main" Nov 03 20:54:26.566380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566332 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="21519c85-ce0b-436a-b195-a454392392e9" containerName="revoke-keys" Nov 03 20:54:26.566380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566340 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" Nov 03 20:54:26.566380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.566349 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0496e47-f1e4-4d10-b865-787fe8338fb7" containerName="main" Nov 03 20:54:26.571870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.571848 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.578817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.578794 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:26.582842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.582728 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:54:26.597887 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.597855 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn"] Nov 03 20:54:26.638202 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.638172 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswm8\" (UniqueName: \"kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.638406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.638380 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.638596 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.638550 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.740051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.739997 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswm8\" (UniqueName: \"kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.740242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.740142 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.740242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.740215 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.740817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.740796 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.742889 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.742867 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.752529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.752495 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswm8\" (UniqueName: \"kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8\") pod \"maas-api-revoke-sub-64c12381-64zbn\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:26.909267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:26.909188 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:27.208040 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:54:27.208008 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6cdf8c_3a4c_4b0b_8af5_fd6d1196ebc1.slice/crio-7bdefd21857fbed6b0a281b93fb90be9a2b83a1cc3c734e20977a4fc958e2f87 WatchSource:0}: Error finding container 7bdefd21857fbed6b0a281b93fb90be9a2b83a1cc3c734e20977a4fc958e2f87: Status 404 returned error can't find the container with id 7bdefd21857fbed6b0a281b93fb90be9a2b83a1cc3c734e20977a4fc958e2f87 Nov 03 20:54:27.212270 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:27.212246 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn"] Nov 03 20:54:27.800150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:27.800042 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" event={"ID":"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1","Type":"ContainerStarted","Data":"c73b9659726d44dd5b70826bbad8f94916b3a13b06137e8d99b59347bad351c1"} Nov 03 20:54:27.800940 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:27.800915 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" event={"ID":"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1","Type":"ContainerStarted","Data":"7bdefd21857fbed6b0a281b93fb90be9a2b83a1cc3c734e20977a4fc958e2f87"} Nov 03 20:54:27.820864 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:27.820820 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" podStartSLOduration=1.820807092 podStartE2EDuration="1.820807092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:27.818478795 +0000 UTC m=+3177.348824468" watchObservedRunningTime="2026-11-03 20:54:27.820807092 +0000 UTC m=+3177.351152776" Nov 03 20:54:28.811617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:28.811580 5815 generic.go:356] "Generic (PLEG): container finished" podID="ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" containerID="c73b9659726d44dd5b70826bbad8f94916b3a13b06137e8d99b59347bad351c1" exitCode=0 Nov 03 20:54:28.812125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:28.811663 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" event={"ID":"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1","Type":"ContainerDied","Data":"c73b9659726d44dd5b70826bbad8f94916b3a13b06137e8d99b59347bad351c1"} Nov 03 20:54:30.091630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.091568 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:30.229175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.229136 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" podUID="f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" containerName="payload-processing" containerID="cri-o://decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf" gracePeriod=25 Nov 03 20:54:30.247049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.247010 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerName="payload-pre-processing" containerID="cri-o://784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962" gracePeriod=25 Nov 03 20:54:30.252974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.252950 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca\") pod \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " Nov 03 20:54:30.253104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.253035 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca\") pod \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " Nov 03 20:54:30.253104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.253089 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8\" (UniqueName: \"kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8\") pod \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\" (UID: \"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1\") " Nov 03 20:54:30.253349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.253325 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca" pod "ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" (UID: "ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:30.253928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.253905 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:30.255176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.255158 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca" pod "ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" (UID: "ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:30.255301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.255287 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8" pod "ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" (UID: "ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1"). InnerVolumeSpecName "kube-api-access-bswm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:30.355988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.355962 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:30.355988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.355987 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bswm8\" (UniqueName: \"kubernetes.io/projected/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1-kube-api-access-bswm8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:30.536175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.536148 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:54:30.559700 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.559668 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume\") pod \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " Nov 03 20:54:30.559892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.559773 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p\" (UniqueName: \"kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p\") pod \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\" (UID: \"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac\") " Nov 03 20:54:30.560377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.560333 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume" pod "f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" (UID: "f2ae5a97-8f71-444e-a2b2-aaaa1e441cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:30.563206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.562914 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p" pod "f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" (UID: "f2ae5a97-8f71-444e-a2b2-aaaa1e441cac"). InnerVolumeSpecName "kube-api-access-w9t9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:30.679533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.664312 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9t9p\" (UniqueName: \"kubernetes.io/projected/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-kube-api-access-w9t9p\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:30.679533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.664357 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:30.833836 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.833804 5815 generic.go:356] "Generic (PLEG): container finished" podID="f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" containerID="decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf" exitCode=0 Nov 03 20:54:30.834011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.833862 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" Nov 03 20:54:30.834011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.833884 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" event={"ID":"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac","Type":"ContainerDied","Data":"decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf"} Nov 03 20:54:30.834011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.833941 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q" event={"ID":"f2ae5a97-8f71-444e-a2b2-aaaa1e441cac","Type":"ContainerDied","Data":"2633281a9f6ff135a6e1ba31ae46cf2300fa2dbae68b9c3f505136f7531c904d"} Nov 03 20:54:30.834011 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.833958 5815 scope.go:122] "RemoveContainer" containerID="decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf" Nov 03 20:54:30.835995 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.835970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" event={"ID":"ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1","Type":"ContainerDied","Data":"7bdefd21857fbed6b0a281b93fb90be9a2b83a1cc3c734e20977a4fc958e2f87"} Nov 03 20:54:30.836148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.836000 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn" Nov 03 20:54:30.836148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.836013 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bdefd21857fbed6b0a281b93fb90be9a2b83a1cc3c734e20977a4fc958e2f87" Nov 03 20:54:30.848953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.848930 5815 scope.go:122] "RemoveContainer" containerID="decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf" Nov 03 20:54:30.849249 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:30.849225 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf\": container with ID starting with decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf not found: ID does not exist" containerID="decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf" Nov 03 20:54:30.849347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.849259 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf"} err="failed to get container status \"decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf\": rpc error: code = NotFound desc = could not find container \"decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf\": container with ID starting with decfbf4f7da07ac5ed6d641c5111e2357a0424ecea8f4ae3e302f6702c3080bf not found: ID does not exist" Nov 03 20:54:30.856827 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.856771 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q"] Nov 03 20:54:30.867281 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:30.867261 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d7c7fd-79cb4b758d-lvs8q"] Nov 03 20:54:32.587879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:32.587834 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" path="/var/lib/kubelet/pods/f2ae5a97-8f71-444e-a2b2-aaaa1e441cac/volumes" Nov 03 20:54:38.927863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:38.927822 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:54:39.592412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.592379 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9"] Nov 03 20:54:39.593188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593167 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" containerName="payload-processing" Nov 03 20:54:39.593188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593185 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" containerName="payload-processing" Nov 03 20:54:39.593188 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593194 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" containerName="revoke-keys" Nov 03 20:54:39.593457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593199 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" containerName="revoke-keys" Nov 03 20:54:39.593457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593315 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a47b882-1e6c-4aef-8b71-9e1b8629b857" containerName="istio-proxy" Nov 03 20:54:39.593457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593326 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ae5a97-8f71-444e-a2b2-aaaa1e441cac" containerName="payload-processing" Nov 03 20:54:39.593457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.593333 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" containerName="revoke-keys" Nov 03 20:54:39.598898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.598873 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.603650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.603627 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:54:39.607579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.607542 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9"] Nov 03 20:54:39.643890 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.643865 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.644224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.644202 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.644430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.644402 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.703892 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.703862 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:39.746141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.746094 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.746337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.746320 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.746451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.746435 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.747160 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.747114 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.749191 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.749165 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.760561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.760538 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497\") pod \"maas-api-revoke-sub-940bfa73-t62b9\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:39.930282 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:39.928749 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:40.469532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:40.461704 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9"] Nov 03 20:54:40.959547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:40.959487 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" event={"ID":"0deab220-780b-4987-9c82-66f668773c95","Type":"ContainerStarted","Data":"73c1ad654b0c3bda4fc0cbfdfd578556829128d06d32b776a8d0de1261be5139"} Nov 03 20:54:41.972338 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:41.972300 5815 generic.go:356] "Generic (PLEG): container finished" podID="0deab220-780b-4987-9c82-66f668773c95" containerID="57477604025da5de2e7072e080ae6b317b9ec41c755917ab685fbdeb1979e2d5" exitCode=0 Nov 03 20:54:41.972817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:41.972343 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" event={"ID":"0deab220-780b-4987-9c82-66f668773c95","Type":"ContainerDied","Data":"57477604025da5de2e7072e080ae6b317b9ec41c755917ab685fbdeb1979e2d5"} Nov 03 20:54:43.209019 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.208995 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:43.326329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.326245 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497\") pod \"0deab220-780b-4987-9c82-66f668773c95\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " Nov 03 20:54:43.326329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.326294 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca\") pod \"0deab220-780b-4987-9c82-66f668773c95\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " Nov 03 20:54:43.326593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.326402 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca\") pod \"0deab220-780b-4987-9c82-66f668773c95\" (UID: \"0deab220-780b-4987-9c82-66f668773c95\") " Nov 03 20:54:43.326701 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.326681 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca" pod "0deab220-780b-4987-9c82-66f668773c95" (UID: "0deab220-780b-4987-9c82-66f668773c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:43.327207 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.327140 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0deab220-780b-4987-9c82-66f668773c95-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:43.328664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.328641 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca" pod "0deab220-780b-4987-9c82-66f668773c95" (UID: "0deab220-780b-4987-9c82-66f668773c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:43.328774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.328721 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497" pod "0deab220-780b-4987-9c82-66f668773c95" (UID: "0deab220-780b-4987-9c82-66f668773c95"). InnerVolumeSpecName "kube-api-access-rs497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:43.429027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.428991 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs497\" (UniqueName: \"kubernetes.io/projected/0deab220-780b-4987-9c82-66f668773c95-kube-api-access-rs497\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:43.429027 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:43.429025 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0deab220-780b-4987-9c82-66f668773c95-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:43.854311 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:43.854275 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:54:44.001368 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:44.001338 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" Nov 03 20:54:44.001617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:44.001368 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9" event={"ID":"0deab220-780b-4987-9c82-66f668773c95","Type":"ContainerDied","Data":"73c1ad654b0c3bda4fc0cbfdfd578556829128d06d32b776a8d0de1261be5139"} Nov 03 20:54:44.001617 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:44.001421 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c1ad654b0c3bda4fc0cbfdfd578556829128d06d32b776a8d0de1261be5139" Nov 03 20:54:48.440315 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:48.440279 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m"] Nov 03 20:54:48.440990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:48.440587 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" containerID="cri-o://16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7" gracePeriod=30 Nov 03 20:54:48.928079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:48.928039 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:54:50.764386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.764349 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk"] Nov 03 20:54:50.844876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.844845 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj"] Nov 03 20:54:50.853552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.846309 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0deab220-780b-4987-9c82-66f668773c95" containerName="revoke-keys" Nov 03 20:54:50.853764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.853745 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0deab220-780b-4987-9c82-66f668773c95" containerName="revoke-keys" Nov 03 20:54:50.854035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.854023 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0deab220-780b-4987-9c82-66f668773c95" containerName="revoke-keys" Nov 03 20:54:50.858443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.858411 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj"] Nov 03 20:54:50.859366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.858574 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:50.865723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.865698 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:54:50.870377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.869953 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:50.991582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.987286 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzjk\" (UniqueName: \"kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:50.991582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.987462 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:50.991582 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:50.987626 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.091159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.089331 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.091159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.089661 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzjk\" (UniqueName: \"kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.091159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.089855 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.091159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.090072 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.099567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.096581 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.122630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.122597 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzjk\" (UniqueName: \"kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk\") pod \"maas-api-revoke-sub-eb19d410-tctqj\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.156248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.156219 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:54:51.192379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.190645 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume\") pod \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " Nov 03 20:54:51.192379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.190944 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b\" (UniqueName: \"kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b\") pod \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\" (UID: \"7b8726ff-df3c-4b16-a272-5567d6f38b3a\") " Nov 03 20:54:51.192379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.191384 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume" pod "7b8726ff-df3c-4b16-a272-5567d6f38b3a" (UID: "7b8726ff-df3c-4b16-a272-5567d6f38b3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:51.192379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.191740 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8726ff-df3c-4b16-a272-5567d6f38b3a-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:51.196528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.196190 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b" pod "7b8726ff-df3c-4b16-a272-5567d6f38b3a" (UID: "7b8726ff-df3c-4b16-a272-5567d6f38b3a"). InnerVolumeSpecName "kube-api-access-42w6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:51.201563 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.199376 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:51.298620 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.295569 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42w6b\" (UniqueName: \"kubernetes.io/projected/7b8726ff-df3c-4b16-a272-5567d6f38b3a-kube-api-access-42w6b\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:51.458535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.451081 5815 generic.go:356] "Generic (PLEG): container finished" podID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerID="784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962" exitCode=0 Nov 03 20:54:51.458535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.451258 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" event={"ID":"7b8726ff-df3c-4b16-a272-5567d6f38b3a","Type":"ContainerDied","Data":"784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962"} Nov 03 20:54:51.458535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.451353 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" event={"ID":"7b8726ff-df3c-4b16-a272-5567d6f38b3a","Type":"ContainerDied","Data":"cc8ffb1319354f472fa5ddb10a25d39a7135fcec80970a49a9e09b18209c0769"} Nov 03 20:54:51.458535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.451384 5815 scope.go:122] "RemoveContainer" containerID="784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962" Nov 03 20:54:51.458535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.451585 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7" Nov 03 20:54:51.477604 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.477571 5815 scope.go:122] "RemoveContainer" containerID="784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962" Nov 03 20:54:51.478082 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:51.477984 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962\": container with ID starting with 784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962 not found: ID does not exist" containerID="784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962" Nov 03 20:54:51.478082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.478023 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962"} err="failed to get container status \"784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962\": rpc error: code = NotFound desc = could not find container \"784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962\": container with ID starting with 784c0c7ccaf3c47dc54c2b674789a7fbd6ab50e20143b78ddcad33d687e52962 not found: ID does not exist" Nov 03 20:54:51.495644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.492718 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7"] Nov 03 20:54:51.503641 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.501753 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d7c7fd-5797dbc64c-mb9x7"] Nov 03 20:54:51.527618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:51.527587 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj"] Nov 03 20:54:52.463159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:52.463123 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" event={"ID":"c09163f3-ab5c-44d7-a762-e195fa429018","Type":"ContainerStarted","Data":"44f483be7a55e6418c8ed40bfc92fb539c718684bc8dbd1a74f4407a666564df"} Nov 03 20:54:52.463698 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:52.463188 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" event={"ID":"c09163f3-ab5c-44d7-a762-e195fa429018","Type":"ContainerStarted","Data":"106850dd4a1a0e5b09a98dff7556a070a9d7f8f7e417a46b9b9a4425dcd85647"} Nov 03 20:54:52.486605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:52.486554 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" podStartSLOduration=2.48653976 podStartE2EDuration="2.48653976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:52.482243933 +0000 UTC m=+3202.012589606" watchObservedRunningTime="2026-11-03 20:54:52.48653976 +0000 UTC m=+3202.016885426" Nov 03 20:54:52.593314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:52.593277 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" path="/var/lib/kubelet/pods/7b8726ff-df3c-4b16-a272-5567d6f38b3a/volumes" Nov 03 20:54:53.481434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.481405 5815 generic.go:356] "Generic (PLEG): container finished" podID="c09163f3-ab5c-44d7-a762-e195fa429018" containerID="44f483be7a55e6418c8ed40bfc92fb539c718684bc8dbd1a74f4407a666564df" exitCode=0 Nov 03 20:54:53.481912 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.481457 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" event={"ID":"c09163f3-ab5c-44d7-a762-e195fa429018","Type":"ContainerDied","Data":"44f483be7a55e6418c8ed40bfc92fb539c718684bc8dbd1a74f4407a666564df"} Nov 03 20:54:53.829557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.829485 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:54:53.966130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966093 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966139 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966337 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966407 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966374 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966471 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966609 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:53.966662 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966634 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:53.966773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966681 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966736 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966886 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966784 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.966947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966894 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:53.966947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.966910 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data\") pod \"904186e6-a013-4ce4-ab77-0d5893e2df5e\" (UID: \"904186e6-a013-4ce4-ab77-0d5893e2df5e\") " Nov 03 20:54:53.967104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967081 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:53.967344 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967313 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:53.967854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967827 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.967854 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967856 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.968061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967870 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.968061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967883 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.968061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.967896 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904186e6-a013-4ce4-ab77-0d5893e2df5e-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.968955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.968933 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:54:53.969460 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.969428 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:54:53.969766 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.969737 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "kube-api-access-8mtdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:53.970233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:53.970213 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token" pod "904186e6-a013-4ce4-ab77-0d5893e2df5e" (UID: "904186e6-a013-4ce4-ab77-0d5893e2df5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:54.070060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.069984 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:54.070060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.070022 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:54.070060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.070038 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtdx\" (UniqueName: \"kubernetes.io/projected/904186e6-a013-4ce4-ab77-0d5893e2df5e-kube-api-access-8mtdx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:54.070060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.070053 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904186e6-a013-4ce4-ab77-0d5893e2df5e-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:54.496697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.496655 5815 generic.go:356] "Generic (PLEG): container finished" podID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerID="16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7" exitCode=0 Nov 03 20:54:54.497149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.496885 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" Nov 03 20:54:54.497149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.496896 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" event={"ID":"904186e6-a013-4ce4-ab77-0d5893e2df5e","Type":"ContainerDied","Data":"16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7"} Nov 03 20:54:54.497149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.496961 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m" event={"ID":"904186e6-a013-4ce4-ab77-0d5893e2df5e","Type":"ContainerDied","Data":"fb5a6cf0e24c2f2e6c4ed9869cd5dfbb2351a23b99ce5fad3a3c95185c4e51c4"} Nov 03 20:54:54.497149 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.496991 5815 scope.go:122] "RemoveContainer" containerID="16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7" Nov 03 20:54:54.524752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.524730 5815 scope.go:122] "RemoveContainer" containerID="afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f" Nov 03 20:54:54.547572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.547442 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m"] Nov 03 20:54:54.555655 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.555626 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d7c7fd-openshift-default-658887fd8c-5cn4m"] Nov 03 20:54:54.556114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.556094 5815 scope.go:122] "RemoveContainer" containerID="16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7" Nov 03 20:54:54.556594 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:54.556465 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7\": container with ID starting with 16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7 not found: ID does not exist" containerID="16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7" Nov 03 20:54:54.556594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.556493 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7"} err="failed to get container status \"16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7\": rpc error: code = NotFound desc = could not find container \"16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7\": container with ID starting with 16ed14682571b5578727eafaebb52acbe8769ae9ee271584898ffe90e9cdbdf7 not found: ID does not exist" Nov 03 20:54:54.556594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.556528 5815 scope.go:122] "RemoveContainer" containerID="afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f" Nov 03 20:54:54.557080 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:54:54.556953 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f\": container with ID starting with afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f not found: ID does not exist" containerID="afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f" Nov 03 20:54:54.557080 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.556988 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f"} err="failed to get container status \"afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f\": rpc error: code = NotFound desc = could not find container \"afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f\": container with ID starting with afe480eac57b34ee5599d523312e5152e48ffd7a07ee544fd11fad1abc99eb8f not found: ID does not exist" Nov 03 20:54:54.593782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.593754 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" path="/var/lib/kubelet/pods/904186e6-a013-4ce4-ab77-0d5893e2df5e/volumes" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.649250 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k"] Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650281 5815 cpu_manager.go:414] "containerMap: removing container" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650301 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650321 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerName="payload-pre-processing" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650326 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerName="payload-pre-processing" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650341 5815 cpu_manager.go:414] "containerMap: removing container" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650346 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650533 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650552 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8726ff-df3c-4b16-a272-5567d6f38b3a" containerName="payload-pre-processing" Nov 03 20:54:54.651571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.650563 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="904186e6-a013-4ce4-ab77-0d5893e2df5e" containerName="istio-proxy" Nov 03 20:54:54.657150 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.657123 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k"] Nov 03 20:54:54.657280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.657238 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:54.671182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.664711 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f3a4c29a-dockercfg-mfw52\"" Nov 03 20:54:54.671182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.664829 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f3a4c29a\"" Nov 03 20:54:54.671182 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.665039 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm"] Nov 03 20:54:54.701119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.701098 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm"] Nov 03 20:54:54.701343 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.701207 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:54.968119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.967093 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:54.968119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.967165 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:54.968119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.967355 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llt5\" (UniqueName: \"kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5\") pod \"payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:54.968119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:54.967529 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4j8h\" (UniqueName: \"kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h\") pod \"payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:55.002195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.002162 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:55.069105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.069019 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca\") pod \"c09163f3-ab5c-44d7-a762-e195fa429018\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " Nov 03 20:54:55.069621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.069562 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca\") pod \"c09163f3-ab5c-44d7-a762-e195fa429018\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " Nov 03 20:54:55.069952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.069931 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk\" (UniqueName: \"kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk\") pod \"c09163f3-ab5c-44d7-a762-e195fa429018\" (UID: \"c09163f3-ab5c-44d7-a762-e195fa429018\") " Nov 03 20:54:55.070593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.070547 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4j8h\" (UniqueName: \"kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h\") pod \"payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:55.070831 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.070814 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:55.071029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.071010 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:55.071412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.071361 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llt5\" (UniqueName: \"kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5\") pod \"payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:55.072644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.072621 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca" pod "c09163f3-ab5c-44d7-a762-e195fa429018" (UID: "c09163f3-ab5c-44d7-a762-e195fa429018"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:55.083569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.080336 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume\") pod \"payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:55.083569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.081359 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:55.084297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.084273 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca" pod "c09163f3-ab5c-44d7-a762-e195fa429018" (UID: "c09163f3-ab5c-44d7-a762-e195fa429018"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:55.093608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.093545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llt5\" (UniqueName: \"kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5\") pod \"payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:55.095773 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.094429 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4j8h\" (UniqueName: \"kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h\") pod \"payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:55.103936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.103719 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk" pod "c09163f3-ab5c-44d7-a762-e195fa429018" (UID: "c09163f3-ab5c-44d7-a762-e195fa429018"). InnerVolumeSpecName "kube-api-access-hqzjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:55.174109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.174073 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:55.174109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.174113 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c09163f3-ab5c-44d7-a762-e195fa429018-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:55.174388 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.174131 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqzjk\" (UniqueName: \"kubernetes.io/projected/c09163f3-ab5c-44d7-a762-e195fa429018-kube-api-access-hqzjk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:55.304761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.304726 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:55.329349 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.329321 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:55.527038 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.526992 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" Nov 03 20:54:55.527550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.527026 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj" event={"ID":"c09163f3-ab5c-44d7-a762-e195fa429018","Type":"ContainerDied","Data":"106850dd4a1a0e5b09a98dff7556a070a9d7f8f7e417a46b9b9a4425dcd85647"} Nov 03 20:54:55.527550 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.527088 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106850dd4a1a0e5b09a98dff7556a070a9d7f8f7e417a46b9b9a4425dcd85647" Nov 03 20:54:55.714584 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:54:55.714208 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3671b384_d11a_46bf_aa8d_d52c17b285bc.slice/crio-11e562e0856de49603434d6c2bb299a30a1447e1ce2294195e2c29696583fbcc WatchSource:0}: Error finding container 11e562e0856de49603434d6c2bb299a30a1447e1ce2294195e2c29696583fbcc: Status 404 returned error can't find the container with id 11e562e0856de49603434d6c2bb299a30a1447e1ce2294195e2c29696583fbcc Nov 03 20:54:55.716465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.716439 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm"] Nov 03 20:54:55.801071 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:54:55.801031 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2843ac3_24e8_4885_baf7_b782d700cf3b.slice/crio-ce737818d0ae2510657c27c2aa4f6be9417d20badfc7f6c4f3b1ea462386ea87 WatchSource:0}: Error finding container ce737818d0ae2510657c27c2aa4f6be9417d20badfc7f6c4f3b1ea462386ea87: Status 404 returned error can't find the container with id ce737818d0ae2510657c27c2aa4f6be9417d20badfc7f6c4f3b1ea462386ea87 Nov 03 20:54:55.803811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:55.803783 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k"] Nov 03 20:54:56.538718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.538669 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" event={"ID":"3671b384-d11a-46bf-aa8d-d52c17b285bc","Type":"ContainerStarted","Data":"64826a85d28ac634c21a7fc06e8fbff827a7de2ac5a49f24f176d9ce869a8863"} Nov 03 20:54:56.539215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.538767 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" event={"ID":"3671b384-d11a-46bf-aa8d-d52c17b285bc","Type":"ContainerStarted","Data":"11e562e0856de49603434d6c2bb299a30a1447e1ce2294195e2c29696583fbcc"} Nov 03 20:54:56.539215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.538929 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:54:56.540630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.540601 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" event={"ID":"d2843ac3-24e8-4885-baf7-b782d700cf3b","Type":"ContainerStarted","Data":"d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5"} Nov 03 20:54:56.540780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.540645 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" event={"ID":"d2843ac3-24e8-4885-baf7-b782d700cf3b","Type":"ContainerStarted","Data":"ce737818d0ae2510657c27c2aa4f6be9417d20badfc7f6c4f3b1ea462386ea87"} Nov 03 20:54:56.540780 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.540699 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:54:56.560006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:56.559963 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" podStartSLOduration=2.559951591 podStartE2EDuration="2.559951591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:56.556361096 +0000 UTC m=+3206.086706770" watchObservedRunningTime="2026-11-03 20:54:56.559951591 +0000 UTC m=+3206.090297264" Nov 03 20:54:57.085901 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:57.085834 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" podStartSLOduration=3.08581324 podStartE2EDuration="3.08581324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:56.577925978 +0000 UTC m=+3206.108271651" watchObservedRunningTime="2026-11-03 20:54:57.08581324 +0000 UTC m=+3206.616158913" Nov 03 20:54:57.091477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:54:57.091452 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:55:01.634533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:01.633949 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9"] Nov 03 20:55:01.857548 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:01.857068 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:55:05.828521 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:05.828462 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" podUID="90948292-dd9b-4b36-8608-000229f5c249" containerName="main" containerID="cri-o://7df99da29895a4a47d47ca139832fd077632a8db9dff3d83456faf9617a09e44" gracePeriod=45 Nov 03 20:55:05.955186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:05.955142 5815 generic.go:356] "Generic (PLEG): container finished" podID="90948292-dd9b-4b36-8608-000229f5c249" containerID="7df99da29895a4a47d47ca139832fd077632a8db9dff3d83456faf9617a09e44" exitCode=0 Nov 03 20:55:05.955357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:05.955214 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" event={"ID":"90948292-dd9b-4b36-8608-000229f5c249","Type":"ContainerDied","Data":"7df99da29895a4a47d47ca139832fd077632a8db9dff3d83456faf9617a09e44"} Nov 03 20:55:06.169918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.169890 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:55:06.266724 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.266688 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache\") pod \"90948292-dd9b-4b36-8608-000229f5c249\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " Nov 03 20:55:06.266957 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.266936 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs\" (UniqueName: \"kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs\") pod \"90948292-dd9b-4b36-8608-000229f5c249\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " Nov 03 20:55:06.267048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.266959 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache" pod "90948292-dd9b-4b36-8608-000229f5c249" (UID: "90948292-dd9b-4b36-8608-000229f5c249"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:06.267104 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.267062 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir\") pod \"90948292-dd9b-4b36-8608-000229f5c249\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " Nov 03 20:55:06.267163 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.267118 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm\") pod \"90948292-dd9b-4b36-8608-000229f5c249\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " Nov 03 20:55:06.267246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.267212 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8\" (UniqueName: \"kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8\") pod \"90948292-dd9b-4b36-8608-000229f5c249\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " Nov 03 20:55:06.267314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.267252 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home\") pod \"90948292-dd9b-4b36-8608-000229f5c249\" (UID: \"90948292-dd9b-4b36-8608-000229f5c249\") " Nov 03 20:55:06.267314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.267296 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir" pod "90948292-dd9b-4b36-8608-000229f5c249" (UID: "90948292-dd9b-4b36-8608-000229f5c249"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:06.267609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.267586 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home" pod "90948292-dd9b-4b36-8608-000229f5c249" (UID: "90948292-dd9b-4b36-8608-000229f5c249"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:06.268265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.268242 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.268383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.268271 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.268383 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.268286 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.270020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.269984 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs" pod "90948292-dd9b-4b36-8608-000229f5c249" (UID: "90948292-dd9b-4b36-8608-000229f5c249"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:06.270020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.270019 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm" pod "90948292-dd9b-4b36-8608-000229f5c249" (UID: "90948292-dd9b-4b36-8608-000229f5c249"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:06.270212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.270093 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8" pod "90948292-dd9b-4b36-8608-000229f5c249" (UID: "90948292-dd9b-4b36-8608-000229f5c249"). InnerVolumeSpecName "kube-api-access-dznc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:06.369392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.369364 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90948292-dd9b-4b36-8608-000229f5c249-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.369392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.369389 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90948292-dd9b-4b36-8608-000229f5c249-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.369650 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.369401 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dznc8\" (UniqueName: \"kubernetes.io/projected/90948292-dd9b-4b36-8608-000229f5c249-kube-api-access-dznc8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:06.967201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.967160 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" event={"ID":"90948292-dd9b-4b36-8608-000229f5c249","Type":"ContainerDied","Data":"39d3e1ccd7d3e517d5f0ebd97a6f7254e5b5f9f25ff9b70d8034df1997deff5f"} Nov 03 20:55:06.967657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.967220 5815 scope.go:122] "RemoveContainer" containerID="7df99da29895a4a47d47ca139832fd077632a8db9dff3d83456faf9617a09e44" Nov 03 20:55:06.967657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.967384 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk" Nov 03 20:55:06.990072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.990042 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk"] Nov 03 20:55:06.996171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:06.996149 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4d63cd/test-model-4d63cd-kserve-8455dfbd5f-htfwk"] Nov 03 20:55:07.559116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:07.559087 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:55:07.559297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:07.559151 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:55:08.658531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:08.657736 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90948292-dd9b-4b36-8608-000229f5c249" path="/var/lib/kubelet/pods/90948292-dd9b-4b36-8608-000229f5c249/volumes" Nov 03 20:55:13.857956 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:55:13.857917 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:55:17.757657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:17.757596 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" podUID="c34b18ef-3f1a-4291-849f-8a297b233b93" containerName="main" containerID="cri-o://0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87" gracePeriod=44 Nov 03 20:55:18.147254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.147217 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:55:18.173386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.173344 5815 generic.go:356] "Generic (PLEG): container finished" podID="c34b18ef-3f1a-4291-849f-8a297b233b93" containerID="0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87" exitCode=0 Nov 03 20:55:18.173571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.173479 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" event={"ID":"c34b18ef-3f1a-4291-849f-8a297b233b93","Type":"ContainerDied","Data":"0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87"} Nov 03 20:55:18.173571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.173535 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" event={"ID":"c34b18ef-3f1a-4291-849f-8a297b233b93","Type":"ContainerDied","Data":"86979c7678906672b9f7356455554d18226a5a2ba44f47dd00535f1aef0d2c36"} Nov 03 20:55:18.173571 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.173567 5815 scope.go:122] "RemoveContainer" containerID="0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87" Nov 03 20:55:18.173800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.173782 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9" Nov 03 20:55:18.202280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.202253 5815 scope.go:122] "RemoveContainer" containerID="0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87" Nov 03 20:55:18.202782 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:55:18.202756 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87\": container with ID starting with 0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87 not found: ID does not exist" containerID="0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87" Nov 03 20:55:18.202899 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.202788 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87"} err="failed to get container status \"0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87\": rpc error: code = NotFound desc = could not find container \"0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87\": container with ID starting with 0baa7c3ab6aedbc6d65119ff3b6114b6de28c3da096e5b7326d70692e8feee87 not found: ID does not exist" Nov 03 20:55:18.213346 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213318 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache\") pod \"c34b18ef-3f1a-4291-849f-8a297b233b93\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " Nov 03 20:55:18.213472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213407 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm\") pod \"c34b18ef-3f1a-4291-849f-8a297b233b93\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " Nov 03 20:55:18.213570 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213555 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs\") pod \"c34b18ef-3f1a-4291-849f-8a297b233b93\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " Nov 03 20:55:18.213636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213592 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache" pod "c34b18ef-3f1a-4291-849f-8a297b233b93" (UID: "c34b18ef-3f1a-4291-849f-8a297b233b93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:18.213709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213695 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home\") pod \"c34b18ef-3f1a-4291-849f-8a297b233b93\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " Nov 03 20:55:18.213771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213759 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir\") pod \"c34b18ef-3f1a-4291-849f-8a297b233b93\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " Nov 03 20:55:18.213959 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213938 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home" pod "c34b18ef-3f1a-4291-849f-8a297b233b93" (UID: "c34b18ef-3f1a-4291-849f-8a297b233b93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:18.214051 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.213964 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m\" (UniqueName: \"kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m\") pod \"c34b18ef-3f1a-4291-849f-8a297b233b93\" (UID: \"c34b18ef-3f1a-4291-849f-8a297b233b93\") " Nov 03 20:55:18.214121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.214060 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir" pod "c34b18ef-3f1a-4291-849f-8a297b233b93" (UID: "c34b18ef-3f1a-4291-849f-8a297b233b93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:18.214767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.214743 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:18.214906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.214892 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:18.215021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.215008 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:18.217944 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.217862 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs" pod "c34b18ef-3f1a-4291-849f-8a297b233b93" (UID: "c34b18ef-3f1a-4291-849f-8a297b233b93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:18.218064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.218046 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm" pod "c34b18ef-3f1a-4291-849f-8a297b233b93" (UID: "c34b18ef-3f1a-4291-849f-8a297b233b93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:18.220265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.220239 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m" pod "c34b18ef-3f1a-4291-849f-8a297b233b93" (UID: "c34b18ef-3f1a-4291-849f-8a297b233b93"). InnerVolumeSpecName "kube-api-access-l6h8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:18.316644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.316611 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6h8m\" (UniqueName: \"kubernetes.io/projected/c34b18ef-3f1a-4291-849f-8a297b233b93-kube-api-access-l6h8m\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:18.316644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.316640 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c34b18ef-3f1a-4291-849f-8a297b233b93-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:18.316644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.316650 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c34b18ef-3f1a-4291-849f-8a297b233b93-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:18.500670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.500633 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9"] Nov 03 20:55:18.506299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.506269 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-519a78/test-model-519a78-kserve-bfcf7fc57-vk4x9"] Nov 03 20:55:18.590630 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:18.590597 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34b18ef-3f1a-4291-849f-8a297b233b93" path="/var/lib/kubelet/pods/c34b18ef-3f1a-4291-849f-8a297b233b93/volumes" Nov 03 20:55:22.743137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:22.743100 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8"] Nov 03 20:55:22.744266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:22.744231 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" containerID="cri-o://04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95" gracePeriod=30 Nov 03 20:55:28.056811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.056779 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:55:28.228314 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228279 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.228492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228362 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.228492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228440 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.228492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228487 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.228731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228578 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:28.228731 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228707 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.228852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228762 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.228852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228796 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:28.228852 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228800 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.229006 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.228958 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.229075 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229045 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:28.229144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229073 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data\") pod \"e6fc1210-f8a1-49e7-8302-c92c3832919d\" (UID: \"e6fc1210-f8a1-49e7-8302-c92c3832919d\") " Nov 03 20:55:28.229144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229088 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:28.229394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229367 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:28.229949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229926 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.230060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229952 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc1210-f8a1-49e7-8302-c92c3832919d-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.230060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229967 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.230060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229980 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.230060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.229993 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.231119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.231096 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:28.231287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.231265 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:28.231414 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.231398 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:28.231537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.231488 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd" pod "e6fc1210-f8a1-49e7-8302-c92c3832919d" (UID: "e6fc1210-f8a1-49e7-8302-c92c3832919d"). InnerVolumeSpecName "kube-api-access-drhvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:28.282091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.281090 5815 generic.go:356] "Generic (PLEG): container finished" podID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerID="04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95" exitCode=0 Nov 03 20:55:28.282091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.281135 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" event={"ID":"e6fc1210-f8a1-49e7-8302-c92c3832919d","Type":"ContainerDied","Data":"04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95"} Nov 03 20:55:28.282091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.281225 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" event={"ID":"e6fc1210-f8a1-49e7-8302-c92c3832919d","Type":"ContainerDied","Data":"00ff54c0a7ddd9a807f53ddea5259e48ff086905229383f351e09d45a7834332"} Nov 03 20:55:28.282091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.281263 5815 scope.go:122] "RemoveContainer" containerID="04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95" Nov 03 20:55:28.282091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.281554 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8" Nov 03 20:55:28.297800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.297780 5815 scope.go:122] "RemoveContainer" containerID="5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9" Nov 03 20:55:28.310203 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.310184 5815 scope.go:122] "RemoveContainer" containerID="04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95" Nov 03 20:55:28.310678 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:55:28.310495 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95\": container with ID starting with 04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95 not found: ID does not exist" containerID="04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95" Nov 03 20:55:28.310678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.310583 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95"} err="failed to get container status \"04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95\": rpc error: code = NotFound desc = could not find container \"04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95\": container with ID starting with 04fca91d37cac42cb07c2b123fbcf30d07fed7a7789a70b82a834d62e262fe95 not found: ID does not exist" Nov 03 20:55:28.310678 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.310607 5815 scope.go:122] "RemoveContainer" containerID="5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9" Nov 03 20:55:28.310977 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:55:28.310899 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9\": container with ID starting with 5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9 not found: ID does not exist" containerID="5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9" Nov 03 20:55:28.310977 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.310925 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9"} err="failed to get container status \"5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9\": rpc error: code = NotFound desc = could not find container \"5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9\": container with ID starting with 5412f013696a0f1d0cf830b46146d27a3dace74f33a7ad6e951d7d4bf5cf44a9 not found: ID does not exist" Nov 03 20:55:28.314105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.314083 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8"] Nov 03 20:55:28.319711 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.319686 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-hxpl8"] Nov 03 20:55:28.331488 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.331465 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drhvd\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-kube-api-access-drhvd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.331590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.331489 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.331590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.331500 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.331590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.331529 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc1210-f8a1-49e7-8302-c92c3832919d-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:28.652020 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:28.651926 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" path="/var/lib/kubelet/pods/e6fc1210-f8a1-49e7-8302-c92c3832919d/volumes" Nov 03 20:55:34.026412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:34.026376 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf"] Nov 03 20:55:34.031058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:34.031032 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb16cb06-qjwnf"] Nov 03 20:55:34.590039 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:34.590003 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fda728-89ef-4bf5-ad38-55fcbc847d2a" path="/var/lib/kubelet/pods/94fda728-89ef-4bf5-ad38-55fcbc847d2a/volumes" Nov 03 20:55:43.854943 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:55:43.854855 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:55:47.025293 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:47.025260 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr"] Nov 03 20:55:47.029898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:47.029875 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ec8b2d0-t6wxr"] Nov 03 20:55:48.589221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:48.589184 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e385ddf-104c-46f9-8aea-0fbb0aa782fb" path="/var/lib/kubelet/pods/1e385ddf-104c-46f9-8aea-0fbb0aa782fb/volumes" Nov 03 20:55:53.025033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.024997 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw"] Nov 03 20:55:53.029431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.029407 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb18a7da-wsslw"] Nov 03 20:55:53.962052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.962017 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb"] Nov 03 20:55:53.963209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963184 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c09163f3-ab5c-44d7-a762-e195fa429018" containerName="revoke-keys" Nov 03 20:55:53.963209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963206 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c09163f3-ab5c-44d7-a762-e195fa429018" containerName="revoke-keys" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963237 5815 cpu_manager.go:414] "containerMap: removing container" podUID="90948292-dd9b-4b36-8608-000229f5c249" containerName="main" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963245 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90948292-dd9b-4b36-8608-000229f5c249" containerName="main" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963261 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963269 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963283 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963291 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963307 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c34b18ef-3f1a-4291-849f-8a297b233b93" containerName="main" Nov 03 20:55:53.963422 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963314 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34b18ef-3f1a-4291-849f-8a297b233b93" containerName="main" Nov 03 20:55:53.963918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963542 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c09163f3-ab5c-44d7-a762-e195fa429018" containerName="revoke-keys" Nov 03 20:55:53.963918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963558 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="90948292-dd9b-4b36-8608-000229f5c249" containerName="main" Nov 03 20:55:53.963918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963569 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" Nov 03 20:55:53.963918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963581 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fc1210-f8a1-49e7-8302-c92c3832919d" containerName="istio-proxy" Nov 03 20:55:53.963918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.963590 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34b18ef-3f1a-4291-849f-8a297b233b93" containerName="main" Nov 03 20:55:53.984796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.984768 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:53.991065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.990979 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"kube-root-ca.crt\"" Nov 03 20:55:53.992357 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.992330 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"default-dockercfg-pr7n2\"" Nov 03 20:55:53.992498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.992425 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"rate-test-model-ba89e0-kserve-self-signed-certs\"" Nov 03 20:55:53.992498 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.992484 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"openshift-service-ca.crt\"" Nov 03 20:55:53.993495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:53.993463 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb"] Nov 03 20:55:54.084103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.084062 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.084569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.084156 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.084569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.084285 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.086690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.086657 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.086791 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.086737 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.086844 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.086796 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxn5\" (UniqueName: \"kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.194820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.194664 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.195037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.194902 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.195037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.194958 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.195037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.195012 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxn5\" (UniqueName: \"kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.195218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.195082 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.195218 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.195140 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.196265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.195713 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.209530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.196762 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.209530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.197316 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.230928 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.230861 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.234228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.234202 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.246530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.243085 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxn5\" (UniqueName: \"kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5\") pod \"rate-test-model-ba89e0-kserve-dffdb7669-6bfkb\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.320298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.320265 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:55:54.633342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.633261 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6febb5d1-e3ab-4d0e-911e-1aca297896d4" path="/var/lib/kubelet/pods/6febb5d1-e3ab-4d0e-911e-1aca297896d4/volumes" Nov 03 20:55:54.679109 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:55:54.679080 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6772c2ad_ac79_45ea_bf6a_e63c817da3e8.slice/crio-b4110352f533e34f7cc98df99cc1f926e877ce0ec3cfac48d61f95c7a3fec52a WatchSource:0}: Error finding container b4110352f533e34f7cc98df99cc1f926e877ce0ec3cfac48d61f95c7a3fec52a: Status 404 returned error can't find the container with id b4110352f533e34f7cc98df99cc1f926e877ce0ec3cfac48d61f95c7a3fec52a Nov 03 20:55:54.679272 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.679169 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb"] Nov 03 20:55:54.735594 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:54.735565 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" event={"ID":"6772c2ad-ac79-45ea-bf6a-e63c817da3e8","Type":"ContainerStarted","Data":"b4110352f533e34f7cc98df99cc1f926e877ce0ec3cfac48d61f95c7a3fec52a"} Nov 03 20:55:55.748366 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:55.748316 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" event={"ID":"6772c2ad-ac79-45ea-bf6a-e63c817da3e8","Type":"ContainerStarted","Data":"30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2"} Nov 03 20:55:55.774085 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:55.774040 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" podStartSLOduration=2.774026335 podStartE2EDuration="2.774026335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:55.767784479 +0000 UTC m=+3265.298130152" watchObservedRunningTime="2026-11-03 20:55:55.774026335 +0000 UTC m=+3265.304372006" Nov 03 20:55:56.321431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:55:56.321394 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:56:03.025673 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:03.025634 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc"] Nov 03 20:56:03.030233 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:03.030201 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18024fb6-ffmbc"] Nov 03 20:56:04.322073 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:04.322044 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:56:04.334906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:04.334875 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:56:04.605015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:04.604929 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6def77c8-4ce4-4bc5-af6c-599dfe585a00" path="/var/lib/kubelet/pods/6def77c8-4ce4-4bc5-af6c-599dfe585a00/volumes" Nov 03 20:56:04.925621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:04.925542 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:56:07.561317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.561284 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb"] Nov 03 20:56:07.566371 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.566347 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.570896 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.570866 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:07.571045 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.571022 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.571113 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.571083 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.571310 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.571291 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsmc\" (UniqueName: \"kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.575440 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.575418 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb"] Nov 03 20:56:07.577406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.577381 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:56:07.676131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.676068 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsmc\" (UniqueName: \"kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.677744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.677714 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.679297 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.679275 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.692264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.680187 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.705685 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.705655 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsmc\" (UniqueName: \"kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.710034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.710006 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5dba8c-fnmcb\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:07.893744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:07.893663 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:08.199209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:08.199114 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb"] Nov 03 20:56:08.208674 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:56:08.208646 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7067bb27_b8ed_4fe7_b4eb_fc1d2f804155.slice/crio-a667449ac544ee1e01a987b0bc00b226821e42ca3f90017ef6683243748e6541 WatchSource:0}: Error finding container a667449ac544ee1e01a987b0bc00b226821e42ca3f90017ef6683243748e6541: Status 404 returned error can't find the container with id a667449ac544ee1e01a987b0bc00b226821e42ca3f90017ef6683243748e6541 Nov 03 20:56:08.987760 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:08.987723 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" event={"ID":"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155","Type":"ContainerStarted","Data":"d30d952b928ca3828274c30868f11e9ea0186b08bc509a3c59af06e1670fa64b"} Nov 03 20:56:08.988298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:08.987790 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" event={"ID":"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155","Type":"ContainerStarted","Data":"a667449ac544ee1e01a987b0bc00b226821e42ca3f90017ef6683243748e6541"} Nov 03 20:56:09.015789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.015728 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" podStartSLOduration=2.015710837 podStartE2EDuration="2.015710837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:09.009358588 +0000 UTC m=+3278.539704253" watchObservedRunningTime="2026-11-03 20:56:09.015710837 +0000 UTC m=+3278.546056510" Nov 03 20:56:09.919497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.917840 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj"] Nov 03 20:56:09.927143 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.927107 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj"] Nov 03 20:56:09.927322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.927254 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:09.933115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.932792 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"openshift-service-ca.crt\"" Nov 03 20:56:09.933115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.932917 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"default-dockercfg-tzxvn\"" Nov 03 20:56:09.933115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.932984 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"rate-test-model-99a8fe-kserve-self-signed-certs\"" Nov 03 20:56:09.933115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:09.933065 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"kube-root-ca.crt\"" Nov 03 20:56:10.006653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.006610 5815 generic.go:356] "Generic (PLEG): container finished" podID="7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" containerID="d30d952b928ca3828274c30868f11e9ea0186b08bc509a3c59af06e1670fa64b" exitCode=0 Nov 03 20:56:10.007212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.006816 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" event={"ID":"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155","Type":"ContainerDied","Data":"d30d952b928ca3828274c30868f11e9ea0186b08bc509a3c59af06e1670fa64b"} Nov 03 20:56:10.043238 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.043186 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.043431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.043258 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.043431 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.043369 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.043640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.043593 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spn2r\" (UniqueName: \"kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.043856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.043833 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.043961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.043942 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.150531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.147131 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.150531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.147213 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.150531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.147318 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.150531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.147377 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.150531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.147419 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.150531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.147562 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spn2r\" (UniqueName: \"kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.171581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.155992 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.175581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.173928 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.175581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.174837 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.230549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.219452 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spn2r\" (UniqueName: \"kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.230549 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.222857 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.261557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.258694 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs\") pod \"rate-test-model-99a8fe-kserve-64cff5f97c-d69lj\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.550122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.550086 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:10.859052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:10.859019 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj"] Nov 03 20:56:10.859773 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:56:10.859742 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0399410_b41e_446e_8300_22e4486211f7.slice/crio-65e6c3cb5017289c9dd90d550dae6010f6788fb1c2c3cea9ae8dbd2c8c51c995 WatchSource:0}: Error finding container 65e6c3cb5017289c9dd90d550dae6010f6788fb1c2c3cea9ae8dbd2c8c51c995: Status 404 returned error can't find the container with id 65e6c3cb5017289c9dd90d550dae6010f6788fb1c2c3cea9ae8dbd2c8c51c995 Nov 03 20:56:11.021008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.020968 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" event={"ID":"b0399410-b41e-446e-8300-22e4486211f7","Type":"ContainerStarted","Data":"54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71"} Nov 03 20:56:11.021423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.021023 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" event={"ID":"b0399410-b41e-446e-8300-22e4486211f7","Type":"ContainerStarted","Data":"65e6c3cb5017289c9dd90d550dae6010f6788fb1c2c3cea9ae8dbd2c8c51c995"} Nov 03 20:56:11.043779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.043727 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" podStartSLOduration=2.043713398 podStartE2EDuration="2.043713398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:11.040357485 +0000 UTC m=+3280.570703154" watchObservedRunningTime="2026-11-03 20:56:11.043713398 +0000 UTC m=+3280.574059069" Nov 03 20:56:11.297060 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.297032 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:11.417434 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.417397 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca\") pod \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " Nov 03 20:56:11.417676 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.417549 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca\") pod \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " Nov 03 20:56:11.417807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.417782 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc\" (UniqueName: \"kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc\") pod \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\" (UID: \"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155\") " Nov 03 20:56:11.417971 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.417948 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca" pod "7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" (UID: "7067bb27-b8ed-4fe7-b4eb-fc1d2f804155"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:11.418441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.418406 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:11.420303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.420266 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc" pod "7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" (UID: "7067bb27-b8ed-4fe7-b4eb-fc1d2f804155"). InnerVolumeSpecName "kube-api-access-mmsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:11.420420 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.420400 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca" pod "7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" (UID: "7067bb27-b8ed-4fe7-b4eb-fc1d2f804155"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:11.520447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.520343 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:11.520447 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.520382 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsmc\" (UniqueName: \"kubernetes.io/projected/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155-kube-api-access-mmsmc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:11.551180 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:11.551149 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:12.031097 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:12.031071 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" Nov 03 20:56:12.031546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:12.031105 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb" event={"ID":"7067bb27-b8ed-4fe7-b4eb-fc1d2f804155","Type":"ContainerDied","Data":"a667449ac544ee1e01a987b0bc00b226821e42ca3f90017ef6683243748e6541"} Nov 03 20:56:12.031546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:12.031177 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a667449ac544ee1e01a987b0bc00b226821e42ca3f90017ef6683243748e6541" Nov 03 20:56:13.893164 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:56:13.893086 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:56:14.035845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:14.035816 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr"] Nov 03 20:56:14.045467 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:14.045431 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9401ec0a-wx5nr"] Nov 03 20:56:14.067418 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:14.067391 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:56:14.589278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:14.589243 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c4cb04-d9fa-42c4-b539-9a97cab3c572" path="/var/lib/kubelet/pods/d7c4cb04-d9fa-42c4-b539-9a97cab3c572/volumes" Nov 03 20:56:20.550943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:20.550907 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:20.563906 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:20.563882 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:21.219295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:21.219260 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:25.024226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:25.024188 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55"] Nov 03 20:56:25.028579 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:25.028554 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774f6799-l7c55"] Nov 03 20:56:26.591224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:26.591189 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453d6f67-f0da-4e5a-aeda-9511329c9ae0" path="/var/lib/kubelet/pods/453d6f67-f0da-4e5a-aeda-9511329c9ae0/volumes" Nov 03 20:56:36.064465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:36.064437 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:56:36.121454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:36.121410 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:56:36.129298 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:36.128825 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 20:56:36.170287 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:36.170259 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 20:56:37.234863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.234822 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" podUID="b0399410-b41e-446e-8300-22e4486211f7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.242246 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.242213 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 20:56:37.368776 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.368463 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" probeResult="failure" output="Get \"https://10.132.1.146:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.369025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.368998 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.500670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.495610 5815 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.604471 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.604428 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.754588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.752606 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:56:37.754588 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.752673 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.775531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.768626 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": context deadline exceeded" Nov 03 20:56:37.775531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.772601 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" probeResult="failure" output="Get \"https://10.132.1.146:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.846618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.846578 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 20:56:37.848192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.848162 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 20:56:37.890628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.890577 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:37.920709 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:37.919585 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.178686 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.178585 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" probeResult="failure" output="Get \"https://10.132.1.144:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.241007 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.240928 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" podUID="b0399410-b41e-446e-8300-22e4486211f7" containerName="main" probeResult="failure" output="Get \"https://10.132.1.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.245537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.241663 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:38.777547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.769022 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" probeResult="failure" output="Get \"https://10.132.0.110:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.777547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.769588 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:56:38.847580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.847537 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:38.849145 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.849125 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:56:38.915918 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.915862 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:39.041786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:38.920284 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.154:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:39.041786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.009689 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 20:56:39.041786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.040666 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:56:39.041786 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.040733 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:56:39.152054 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.152027 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 20:56:39.194818 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.194777 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 20:56:39.228614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.228582 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 20:56:39.297611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.280989 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 20:56:39.474533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.472677 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:56:39.474533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:39.472747 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:56:43.854624 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:56:43.854585 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:56:46.301958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:46.301925 5815 scope.go:122] "RemoveContainer" containerID="b05a56f9c86f3b31500b1d43f03bd2bcc9106579bc389e36da2f6fec508b1a4b" Nov 03 20:56:46.317647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:56:46.317627 5815 scope.go:122] "RemoveContainer" containerID="7f408ee071a768c9ac40dfecbade0a28859bd1269ecc58f59d3e3363c45b5c99" Nov 03 20:57:07.411306 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:07.411270 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:57:08.958158 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:08.958071 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49"] Nov 03 20:57:08.958770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:08.958376 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" containerID="cri-o://5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5" gracePeriod=30 Nov 03 20:57:11.516789 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:11.516759 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx"] Nov 03 20:57:11.517240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:11.516956 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" containerID="cri-o://47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820" gracePeriod=30 Nov 03 20:57:13.857378 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:13.857329 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:57:14.264539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.264495 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:57:14.373964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.373929 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.373964 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.373977 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374240 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374031 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374246 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374282 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:14.374311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374303 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374365 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374449 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:14.374601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374477 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374522 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374601 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374596 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs\") pod \"275de0e5-b391-42e6-9a41-641ae3e52d3a\" (UID: \"275de0e5-b391-42e6-9a41-641ae3e52d3a\") " Nov 03 20:57:14.374895 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.374800 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:14.375332 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.375027 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:14.375699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.375458 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.375699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.375477 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/275de0e5-b391-42e6-9a41-641ae3e52d3a-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.375699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.375493 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.375699 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.375526 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.376072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.376032 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:14.378239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.378211 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "kube-api-access-9s7vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:14.378352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.378252 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:14.378870 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.378833 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:14.379103 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.379079 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy" pod "275de0e5-b391-42e6-9a41-641ae3e52d3a" (UID: "275de0e5-b391-42e6-9a41-641ae3e52d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:14.476735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.476646 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.476735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.476674 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.476735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.476685 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.476735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.476695 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/275de0e5-b391-42e6-9a41-641ae3e52d3a-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:14.476735 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:14.476711 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7vx\" (UniqueName: \"kubernetes.io/projected/275de0e5-b391-42e6-9a41-641ae3e52d3a-kube-api-access-9s7vx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:15.013546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.013490 5815 generic.go:356] "Generic (PLEG): container finished" podID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerID="5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5" exitCode=0 Nov 03 20:57:15.014064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.013577 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" Nov 03 20:57:15.014064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.013588 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" event={"ID":"275de0e5-b391-42e6-9a41-641ae3e52d3a","Type":"ContainerDied","Data":"5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5"} Nov 03 20:57:15.014064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.013649 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49" event={"ID":"275de0e5-b391-42e6-9a41-641ae3e52d3a","Type":"ContainerDied","Data":"e6a002f3c52c87af4ddbdb54cef93577b461c99558cae32cff7c107e9b506c00"} Nov 03 20:57:15.014064 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.013673 5815 scope.go:122] "RemoveContainer" containerID="5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5" Nov 03 20:57:15.027253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.027236 5815 scope.go:122] "RemoveContainer" containerID="5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5" Nov 03 20:57:15.027568 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:15.027549 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5\": container with ID starting with 5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5 not found: ID does not exist" containerID="5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5" Nov 03 20:57:15.027659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.027577 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5"} err="failed to get container status \"5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5\": rpc error: code = NotFound desc = could not find container \"5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5\": container with ID starting with 5201060add000f655ff6b5402e45cac0ced85b823c21fa815214e189d55894b5 not found: ID does not exist" Nov 03 20:57:15.037642 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.037615 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49"] Nov 03 20:57:15.042165 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.042146 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-flz49"] Nov 03 20:57:15.629799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.629767 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7"] Nov 03 20:57:15.630034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:15.630001 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" containerID="cri-o://0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388" gracePeriod=30 Nov 03 20:57:16.589144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.589113 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" path="/var/lib/kubelet/pods/275de0e5-b391-42e6-9a41-641ae3e52d3a/volumes" Nov 03 20:57:16.837681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.837614 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:57:16.926061 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926026 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926168 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926242 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926282 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926376 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926477 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926555 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926611 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.926800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.926656 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy\") pod \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\" (UID: \"4fb00ef0-8914-4629-a6fe-bf7eee0b3688\") " Nov 03 20:57:16.927295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.927000 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:16.927295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.927188 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:16.932014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.931796 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:16.932014 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.931843 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:16.933775 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.933725 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:16.933961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.933937 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "kube-api-access-64cx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:16.934114 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.934093 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:16.934228 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.934188 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:16.935419 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.935368 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:16.937554 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.932615 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:16.944561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:16.943061 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy" pod "4fb00ef0-8914-4629-a6fe-bf7eee0b3688" (UID: "4fb00ef0-8914-4629-a6fe-bf7eee0b3688"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:17.033364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033334 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.033364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033361 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.033364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033371 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.033652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033381 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cx2\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-kube-api-access-64cx2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.033652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033391 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.033652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033399 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.033652 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.033407 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb00ef0-8914-4629-a6fe-bf7eee0b3688-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.037386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.037352 5815 generic.go:356] "Generic (PLEG): container finished" podID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerID="47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820" exitCode=0 Nov 03 20:57:17.037386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.037375 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" event={"ID":"4fb00ef0-8914-4629-a6fe-bf7eee0b3688","Type":"ContainerDied","Data":"47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820"} Nov 03 20:57:17.037611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.037455 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" Nov 03 20:57:17.037611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.037472 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx" event={"ID":"4fb00ef0-8914-4629-a6fe-bf7eee0b3688","Type":"ContainerDied","Data":"143665b9dbbbae16a01caa108aba07c22e9476930f412600bf8cef5d82a4e05f"} Nov 03 20:57:17.037611 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.037502 5815 scope.go:122] "RemoveContainer" containerID="47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820" Nov 03 20:57:17.050915 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.050902 5815 scope.go:122] "RemoveContainer" containerID="b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b" Nov 03 20:57:17.061545 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.061524 5815 scope.go:122] "RemoveContainer" containerID="47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820" Nov 03 20:57:17.061845 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:17.061825 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820\": container with ID starting with 47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820 not found: ID does not exist" containerID="47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820" Nov 03 20:57:17.061907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.061853 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820"} err="failed to get container status \"47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820\": rpc error: code = NotFound desc = could not find container \"47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820\": container with ID starting with 47c95502d89003b9f22e1769664f8b2497d6ffbdccbb724e687f75ddfbeec820 not found: ID does not exist" Nov 03 20:57:17.061907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.061868 5815 scope.go:122] "RemoveContainer" containerID="b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b" Nov 03 20:57:17.062101 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:17.062085 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b\": container with ID starting with b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b not found: ID does not exist" containerID="b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b" Nov 03 20:57:17.062139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.062108 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b"} err="failed to get container status \"b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b\": rpc error: code = NotFound desc = could not find container \"b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b\": container with ID starting with b969e5e37a55478b454aacd7cd7e715d4d9a583a3207ed50ebf4f85814d5cd4b not found: ID does not exist" Nov 03 20:57:17.072647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.072623 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx"] Nov 03 20:57:17.080074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.080053 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-wnwcx"] Nov 03 20:57:17.314139 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.314101 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k"] Nov 03 20:57:17.319451 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:17.319426 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm"] Nov 03 20:57:18.587477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:18.587444 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" path="/var/lib/kubelet/pods/4fb00ef0-8914-4629-a6fe-bf7eee0b3688/volumes" Nov 03 20:57:19.390072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:19.390025 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9"] Nov 03 20:57:19.390323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:19.390296 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" containerID="cri-o://d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc" gracePeriod=30 Nov 03 20:57:20.939958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:20.939924 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.035354 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.035440 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.035892 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.035952 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.036447 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.036496 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.036580 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.036648 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.036678 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.037661 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.036830 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy\") pod \"346a0e2d-1843-4125-bc93-b6bf25993c3b\" (UID: \"346a0e2d-1843-4125-bc93-b6bf25993c3b\") " Nov 03 20:57:21.042634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.038528 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:21.042634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.038719 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:21.042634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.039227 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:21.042634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.039806 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:21.042634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.041156 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:21.042634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.042317 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "kube-api-access-rwd8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:21.046574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.043341 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:21.046574 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.044682 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token" pod "346a0e2d-1843-4125-bc93-b6bf25993c3b" (UID: "346a0e2d-1843-4125-bc93-b6bf25993c3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:21.121029 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.120997 5815 generic.go:356] "Generic (PLEG): container finished" podID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerID="0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388" exitCode=0 Nov 03 20:57:21.121215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.121098 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" event={"ID":"346a0e2d-1843-4125-bc93-b6bf25993c3b","Type":"ContainerDied","Data":"0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388"} Nov 03 20:57:21.121215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.121110 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" Nov 03 20:57:21.121215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.121163 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7" event={"ID":"346a0e2d-1843-4125-bc93-b6bf25993c3b","Type":"ContainerDied","Data":"f6ea7eb5ec3616a6f0981eff82e36fe4acea6432152c6f8109273a3569a7f32a"} Nov 03 20:57:21.121215 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.121184 5815 scope.go:122] "RemoveContainer" containerID="0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388" Nov 03 20:57:21.136363 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.136340 5815 scope.go:122] "RemoveContainer" containerID="1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3" Nov 03 20:57:21.139429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139407 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139429 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/346a0e2d-1843-4125-bc93-b6bf25993c3b-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139439 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139448 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139458 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwd8z\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-kube-api-access-rwd8z\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139468 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139486 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/346a0e2d-1843-4125-bc93-b6bf25993c3b-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139495 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.139848 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.139522 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/346a0e2d-1843-4125-bc93-b6bf25993c3b-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.152111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.152030 5815 scope.go:122] "RemoveContainer" containerID="0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388" Nov 03 20:57:21.152675 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:21.152649 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388\": container with ID starting with 0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388 not found: ID does not exist" containerID="0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388" Nov 03 20:57:21.152675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.152684 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388"} err="failed to get container status \"0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388\": rpc error: code = NotFound desc = could not find container \"0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388\": container with ID starting with 0112bb514cb7cc82ff9d4a7b82d730bdd5a2725b6db11bb63fd5f816dede3388 not found: ID does not exist" Nov 03 20:57:21.153087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.152708 5815 scope.go:122] "RemoveContainer" containerID="1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3" Nov 03 20:57:21.153087 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:21.153013 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3\": container with ID starting with 1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3 not found: ID does not exist" containerID="1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3" Nov 03 20:57:21.153087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.153044 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3"} err="failed to get container status \"1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3\": rpc error: code = NotFound desc = could not find container \"1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3\": container with ID starting with 1fe4521bdad729aeb772c8fd5867649b3ccfb25227c4cf945ca71a2c9b3c75b3 not found: ID does not exist" Nov 03 20:57:21.157457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.157402 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7"] Nov 03 20:57:21.161624 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:21.161605 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-hdbl7"] Nov 03 20:57:22.314973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:22.314939 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" podUID="d2843ac3-24e8-4885-baf7-b782d700cf3b" containerName="payload-processing" containerID="cri-o://d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5" gracePeriod=25 Nov 03 20:57:22.320475 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:22.320447 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" podUID="3671b384-d11a-46bf-aa8d-d52c17b285bc" containerName="payload-pre-processing" containerID="cri-o://64826a85d28ac634c21a7fc06e8fbff827a7de2ac5a49f24f176d9ce869a8863" gracePeriod=25 Nov 03 20:57:22.923585 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:22.906709 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" path="/var/lib/kubelet/pods/346a0e2d-1843-4125-bc93-b6bf25993c3b/volumes" Nov 03 20:57:23.099148 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.099120 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:57:23.142670 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.142642 5815 generic.go:356] "Generic (PLEG): container finished" podID="3671b384-d11a-46bf-aa8d-d52c17b285bc" containerID="64826a85d28ac634c21a7fc06e8fbff827a7de2ac5a49f24f176d9ce869a8863" exitCode=0 Nov 03 20:57:23.142800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.142720 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" event={"ID":"3671b384-d11a-46bf-aa8d-d52c17b285bc","Type":"ContainerDied","Data":"64826a85d28ac634c21a7fc06e8fbff827a7de2ac5a49f24f176d9ce869a8863"} Nov 03 20:57:23.142800 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.142784 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" event={"ID":"3671b384-d11a-46bf-aa8d-d52c17b285bc","Type":"ContainerDied","Data":"11e562e0856de49603434d6c2bb299a30a1447e1ce2294195e2c29696583fbcc"} Nov 03 20:57:23.142894 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.142808 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11e562e0856de49603434d6c2bb299a30a1447e1ce2294195e2c29696583fbcc" Nov 03 20:57:23.144303 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.144283 5815 generic.go:356] "Generic (PLEG): container finished" podID="d2843ac3-24e8-4885-baf7-b782d700cf3b" containerID="d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5" exitCode=0 Nov 03 20:57:23.144408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.144397 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" event={"ID":"d2843ac3-24e8-4885-baf7-b782d700cf3b","Type":"ContainerDied","Data":"d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5"} Nov 03 20:57:23.144454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.144425 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" event={"ID":"d2843ac3-24e8-4885-baf7-b782d700cf3b","Type":"ContainerDied","Data":"ce737818d0ae2510657c27c2aa4f6be9417d20badfc7f6c4f3b1ea462386ea87"} Nov 03 20:57:23.144454 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.144447 5815 scope.go:122] "RemoveContainer" containerID="d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5" Nov 03 20:57:23.144603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.144588 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k" Nov 03 20:57:23.159299 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.159262 5815 scope.go:122] "RemoveContainer" containerID="d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5" Nov 03 20:57:23.159618 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:23.159596 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5\": container with ID starting with d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5 not found: ID does not exist" containerID="d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5" Nov 03 20:57:23.159618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.159624 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5"} err="failed to get container status \"d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5\": rpc error: code = NotFound desc = could not find container \"d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5\": container with ID starting with d10b04e31b8eda4343b93a281a858220a6d5ec9d40f7e2f9773e2305ea427ac5 not found: ID does not exist" Nov 03 20:57:23.164294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.164268 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5\" (UniqueName: \"kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5\") pod \"d2843ac3-24e8-4885-baf7-b782d700cf3b\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " Nov 03 20:57:23.164396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.164356 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume\") pod \"d2843ac3-24e8-4885-baf7-b782d700cf3b\" (UID: \"d2843ac3-24e8-4885-baf7-b782d700cf3b\") " Nov 03 20:57:23.167093 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.167064 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5" pod "d2843ac3-24e8-4885-baf7-b782d700cf3b" (UID: "d2843ac3-24e8-4885-baf7-b782d700cf3b"). InnerVolumeSpecName "kube-api-access-9llt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:23.167710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.167693 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:57:23.171755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.171733 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume" pod "d2843ac3-24e8-4885-baf7-b782d700cf3b" (UID: "d2843ac3-24e8-4885-baf7-b782d700cf3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:23.265627 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.265596 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h\" (UniqueName: \"kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h\") pod \"3671b384-d11a-46bf-aa8d-d52c17b285bc\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " Nov 03 20:57:23.265797 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.265732 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume\") pod \"3671b384-d11a-46bf-aa8d-d52c17b285bc\" (UID: \"3671b384-d11a-46bf-aa8d-d52c17b285bc\") " Nov 03 20:57:23.266167 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.266142 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume" pod "3671b384-d11a-46bf-aa8d-d52c17b285bc" (UID: "3671b384-d11a-46bf-aa8d-d52c17b285bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:23.266672 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.266650 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3671b384-d11a-46bf-aa8d-d52c17b285bc-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.266779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.266675 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9llt5\" (UniqueName: \"kubernetes.io/projected/d2843ac3-24e8-4885-baf7-b782d700cf3b-kube-api-access-9llt5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.266779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.266691 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2843ac3-24e8-4885-baf7-b782d700cf3b-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.268127 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.268107 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h" pod "3671b384-d11a-46bf-aa8d-d52c17b285bc" (UID: "3671b384-d11a-46bf-aa8d-d52c17b285bc"). InnerVolumeSpecName "kube-api-access-z4j8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:23.368551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.368525 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4j8h\" (UniqueName: \"kubernetes.io/projected/3671b384-d11a-46bf-aa8d-d52c17b285bc-kube-api-access-z4j8h\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.470543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.470497 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k"] Nov 03 20:57:23.476761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:23.476729 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f3a4c29a-846f96f786-f546k"] Nov 03 20:57:24.155025 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.154999 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm" Nov 03 20:57:24.196973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.196947 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm"] Nov 03 20:57:24.243121 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.243088 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f3a4c29a-797bbf59-5cpvm"] Nov 03 20:57:24.600463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.597257 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3671b384-d11a-46bf-aa8d-d52c17b285bc" path="/var/lib/kubelet/pods/3671b384-d11a-46bf-aa8d-d52c17b285bc/volumes" Nov 03 20:57:24.600463 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.598344 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2843ac3-24e8-4885-baf7-b782d700cf3b" path="/var/lib/kubelet/pods/d2843ac3-24e8-4885-baf7-b782d700cf3b/volumes" Nov 03 20:57:24.681074 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.681045 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:57:24.791932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.791900 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.791951 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792020 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792198 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:24.792283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792071 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792270 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:24.792405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792322 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792405 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792384 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792541 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792528 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792585 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.792671 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792634 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert\") pod \"9a087036-27f7-4f16-afc1-e2e4bceb248b\" (UID: \"9a087036-27f7-4f16-afc1-e2e4bceb248b\") " Nov 03 20:57:24.793026 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.792999 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:24.793591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.793557 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.793591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.793584 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.793778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.793599 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.793778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.793623 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:24.793911 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.793778 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:24.796497 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.796475 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:24.796710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.796687 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:24.803551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.798671 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "kube-api-access-46pkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:24.803551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.798757 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo" pod "9a087036-27f7-4f16-afc1-e2e4bceb248b" (UID: "9a087036-27f7-4f16-afc1-e2e4bceb248b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:24.895660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.895623 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.895660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.895656 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.895893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.895671 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.895893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.895686 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a087036-27f7-4f16-afc1-e2e4bceb248b-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.895893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.895700 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pkl\" (UniqueName: \"kubernetes.io/projected/9a087036-27f7-4f16-afc1-e2e4bceb248b-kube-api-access-46pkl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:24.895893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:24.895715 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a087036-27f7-4f16-afc1-e2e4bceb248b-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:25.166249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.166152 5815 generic.go:356] "Generic (PLEG): container finished" podID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerID="d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc" exitCode=0 Nov 03 20:57:25.166249 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.166222 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" Nov 03 20:57:25.166484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.166245 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" event={"ID":"9a087036-27f7-4f16-afc1-e2e4bceb248b","Type":"ContainerDied","Data":"d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc"} Nov 03 20:57:25.166484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.166313 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9" event={"ID":"9a087036-27f7-4f16-afc1-e2e4bceb248b","Type":"ContainerDied","Data":"95bbb6e1bcfa35d08ed5f52a6c5c6cfac6feea9811c48f5cb28f9f8d78a81e59"} Nov 03 20:57:25.166484 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.166343 5815 scope.go:122] "RemoveContainer" containerID="d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc" Nov 03 20:57:25.183491 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.183468 5815 scope.go:122] "RemoveContainer" containerID="d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc" Nov 03 20:57:25.183908 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:25.183875 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc\": container with ID starting with d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc not found: ID does not exist" containerID="d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc" Nov 03 20:57:25.184030 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.183934 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc"} err="failed to get container status \"d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc\": rpc error: code = NotFound desc = could not find container \"d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc\": container with ID starting with d3f8cbc904a5259efa548037a9f1d889165a08bd14eec1566a6029f6452c5ffc not found: ID does not exist" Nov 03 20:57:25.202028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.202001 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9"] Nov 03 20:57:25.223607 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:25.223569 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-zzph9"] Nov 03 20:57:26.606065 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:26.606036 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" path="/var/lib/kubelet/pods/9a087036-27f7-4f16-afc1-e2e4bceb248b/volumes" Nov 03 20:57:33.110529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:33.102266 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92"] Nov 03 20:57:33.133530 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:33.122632 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" containerID="cri-o://6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e" gracePeriod=30 Nov 03 20:57:36.051421 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.051384 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72"] Nov 03 20:57:36.052157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052142 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" Nov 03 20:57:36.052214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052159 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" Nov 03 20:57:36.052214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052185 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3671b384-d11a-46bf-aa8d-d52c17b285bc" containerName="payload-pre-processing" Nov 03 20:57:36.052214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052191 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3671b384-d11a-46bf-aa8d-d52c17b285bc" containerName="payload-pre-processing" Nov 03 20:57:36.052214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052206 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d2843ac3-24e8-4885-baf7-b782d700cf3b" containerName="payload-processing" Nov 03 20:57:36.052214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052210 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2843ac3-24e8-4885-baf7-b782d700cf3b" containerName="payload-processing" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052217 5815 cpu_manager.go:414] "containerMap: removing container" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052222 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052232 5815 cpu_manager.go:414] "containerMap: removing container" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052237 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052244 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052248 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052261 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" containerName="revoke-keys" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052267 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" containerName="revoke-keys" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052375 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052383 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" Nov 03 20:57:36.052387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052390 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a087036-27f7-4f16-afc1-e2e4bceb248b" containerName="istio-proxy" Nov 03 20:57:36.052778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052400 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3671b384-d11a-46bf-aa8d-d52c17b285bc" containerName="payload-pre-processing" Nov 03 20:57:36.052778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052408 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2843ac3-24e8-4885-baf7-b782d700cf3b" containerName="payload-processing" Nov 03 20:57:36.052778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052413 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" Nov 03 20:57:36.052778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052420 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="275de0e5-b391-42e6-9a41-641ae3e52d3a" containerName="istio-proxy" Nov 03 20:57:36.052778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052425 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" Nov 03 20:57:36.052778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.052433 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" containerName="revoke-keys" Nov 03 20:57:36.056178 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.056154 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.061629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.061528 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:36.061629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.061597 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:57:36.137577 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.083485 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72"] Nov 03 20:57:36.138234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.115657 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2k2n\" (UniqueName: \"kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.138565 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.138544 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.138782 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.138763 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.241235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.241100 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.241235 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.241217 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.241593 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.241577 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2k2n\" (UniqueName: \"kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.242190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.242164 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.243804 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.243782 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.253819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.253788 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2k2n\" (UniqueName: \"kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n\") pod \"maas-api-revoke-sub-a0d3c979-qql72\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.435613 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.435520 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:36.766290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:36.765575 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72"] Nov 03 20:57:37.306400 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:37.306370 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2"] Nov 03 20:57:37.306935 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:37.306653 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" containerID="cri-o://65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215" gracePeriod=30 Nov 03 20:57:37.307035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:37.306941 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" event={"ID":"8e1e89ab-2973-4f78-ab89-942ac86351b0","Type":"ContainerStarted","Data":"323139ee6cae8876b130c027a0f4f7bc6d3b96cb2ea3b30b12861525fc289d42"} Nov 03 20:57:37.307035 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:37.306993 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" event={"ID":"8e1e89ab-2973-4f78-ab89-942ac86351b0","Type":"ContainerStarted","Data":"48f0f511be5805d31c2aa1a0ed149424014c7cd13f95a6bbb37f378309227100"} Nov 03 20:57:37.353086 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:37.352999 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" podStartSLOduration=1.352985489 podStartE2EDuration="1.352985489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:37.351899395 +0000 UTC m=+3366.882245070" watchObservedRunningTime="2026-11-03 20:57:37.352985489 +0000 UTC m=+3366.883331208" Nov 03 20:57:38.317529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.317482 5815 generic.go:356] "Generic (PLEG): container finished" podID="8e1e89ab-2973-4f78-ab89-942ac86351b0" containerID="323139ee6cae8876b130c027a0f4f7bc6d3b96cb2ea3b30b12861525fc289d42" exitCode=0 Nov 03 20:57:38.318021 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.317566 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" event={"ID":"8e1e89ab-2973-4f78-ab89-942ac86351b0","Type":"ContainerDied","Data":"323139ee6cae8876b130c027a0f4f7bc6d3b96cb2ea3b30b12861525fc289d42"} Nov 03 20:57:38.531621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.531595 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:57:38.594492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594408 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.594492 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594484 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.594785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594588 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.594785 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594666 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.594910 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594812 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.594974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594908 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.594974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594945 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.595090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.594975 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.595090 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.595049 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk\") pod \"540934a6-2005-4927-b558-441ba575a73d\" (UID: \"540934a6-2005-4927-b558-441ba575a73d\") " Nov 03 20:57:38.596679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.596646 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:38.596979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.596953 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:38.597190 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.597163 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:38.597387 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.597365 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:38.597610 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.597586 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:38.603629 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.603603 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.604008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.603632 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.604008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.603647 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.604008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.603662 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/540934a6-2005-4927-b558-441ba575a73d-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.604008 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.603676 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.609647 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.609619 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:38.609757 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.609729 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:38.609829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.609786 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "kube-api-access-xtjqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:38.609877 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.609829 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token" pod "540934a6-2005-4927-b558-441ba575a73d" (UID: "540934a6-2005-4927-b558-441ba575a73d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:38.705364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.705333 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.705364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.705360 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/540934a6-2005-4927-b558-441ba575a73d-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.705364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.705369 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/540934a6-2005-4927-b558-441ba575a73d-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:38.705653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:38.705379 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtjqk\" (UniqueName: \"kubernetes.io/projected/540934a6-2005-4927-b558-441ba575a73d-kube-api-access-xtjqk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.327771 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.327729 5815 generic.go:356] "Generic (PLEG): container finished" podID="540934a6-2005-4927-b558-441ba575a73d" containerID="6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e" exitCode=0 Nov 03 20:57:39.328261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.327772 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" event={"ID":"540934a6-2005-4927-b558-441ba575a73d","Type":"ContainerDied","Data":"6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e"} Nov 03 20:57:39.328261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.327801 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" Nov 03 20:57:39.328261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.327887 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92" event={"ID":"540934a6-2005-4927-b558-441ba575a73d","Type":"ContainerDied","Data":"422c2ac036aac72042ccf9363eb1dc2fa96edb6116f72dbe84361dac79483061"} Nov 03 20:57:39.328261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.327919 5815 scope.go:122] "RemoveContainer" containerID="6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e" Nov 03 20:57:39.341638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.341619 5815 scope.go:122] "RemoveContainer" containerID="c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb" Nov 03 20:57:39.357489 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.357465 5815 scope.go:122] "RemoveContainer" containerID="6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e" Nov 03 20:57:39.358269 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:39.358233 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e\": container with ID starting with 6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e not found: ID does not exist" containerID="6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e" Nov 03 20:57:39.358374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.358278 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e"} err="failed to get container status \"6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e\": rpc error: code = NotFound desc = could not find container \"6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e\": container with ID starting with 6d762a1a6cb508d6eb6a8ae68a80aa211e6d21ab23412fa019113979aa46e86e not found: ID does not exist" Nov 03 20:57:39.358374 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.358301 5815 scope.go:122] "RemoveContainer" containerID="c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb" Nov 03 20:57:39.358780 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:39.358752 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb\": container with ID starting with c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb not found: ID does not exist" containerID="c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb" Nov 03 20:57:39.358860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.358792 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb"} err="failed to get container status \"c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb\": rpc error: code = NotFound desc = could not find container \"c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb\": container with ID starting with c696c09f5a90558b5bda46b28e6667591857dbabb66db065530e6b0649cdfdcb not found: ID does not exist" Nov 03 20:57:39.363209 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.363183 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92"] Nov 03 20:57:39.369640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.369617 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-7rr92"] Nov 03 20:57:39.533131 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.533106 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:39.646193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.637932 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca\") pod \"8e1e89ab-2973-4f78-ab89-942ac86351b0\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " Nov 03 20:57:39.646193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.638179 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca\") pod \"8e1e89ab-2973-4f78-ab89-942ac86351b0\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " Nov 03 20:57:39.646193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.638261 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n\" (UniqueName: \"kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n\") pod \"8e1e89ab-2973-4f78-ab89-942ac86351b0\" (UID: \"8e1e89ab-2973-4f78-ab89-942ac86351b0\") " Nov 03 20:57:39.661900 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.659839 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca" pod "8e1e89ab-2973-4f78-ab89-942ac86351b0" (UID: "8e1e89ab-2973-4f78-ab89-942ac86351b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:39.673212 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.673159 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca" pod "8e1e89ab-2973-4f78-ab89-942ac86351b0" (UID: "8e1e89ab-2973-4f78-ab89-942ac86351b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:39.675376 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.675349 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n" pod "8e1e89ab-2973-4f78-ab89-942ac86351b0" (UID: "8e1e89ab-2973-4f78-ab89-942ac86351b0"). InnerVolumeSpecName "kube-api-access-x2k2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:39.749640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.749613 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.749640 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.749641 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2k2n\" (UniqueName: \"kubernetes.io/projected/8e1e89ab-2973-4f78-ab89-942ac86351b0-kube-api-access-x2k2n\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.749837 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:39.749651 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e89ab-2973-4f78-ab89-942ac86351b0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:40.382695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:40.382657 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" event={"ID":"8e1e89ab-2973-4f78-ab89-942ac86351b0","Type":"ContainerDied","Data":"48f0f511be5805d31c2aa1a0ed149424014c7cd13f95a6bbb37f378309227100"} Nov 03 20:57:40.383352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:40.383324 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f0f511be5805d31c2aa1a0ed149424014c7cd13f95a6bbb37f378309227100" Nov 03 20:57:40.383635 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:40.383617 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72" Nov 03 20:57:40.589265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:40.589228 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540934a6-2005-4927-b558-441ba575a73d" path="/var/lib/kubelet/pods/540934a6-2005-4927-b558-441ba575a73d/volumes" Nov 03 20:57:42.645109 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.645086 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:57:42.704205 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.704164 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.704496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.704476 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.704695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.704678 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.704817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.704803 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.704992 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.704977 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.705258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.705243 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.705528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.705493 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.707265 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.707242 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.707455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.707430 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs\") pod \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\" (UID: \"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0\") " Nov 03 20:57:42.709050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.708960 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:42.709157 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.709054 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:42.709221 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.709180 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:42.709518 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.709477 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:42.709668 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.709646 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.709750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.709675 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.709750 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.709692 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.710140 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.710102 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:42.715990 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.715951 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:42.716214 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.716187 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "kube-api-access-b4x9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:42.717116 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.717086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:42.717309 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.717281 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo" pod "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" (UID: "2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:42.811325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.811289 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.811325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.811325 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4x9g\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-kube-api-access-b4x9g\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.811540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.811340 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.811540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.811352 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.811540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.811364 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:42.811540 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:42.811375 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:43.219857 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.219821 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5"] Nov 03 20:57:43.220967 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.220937 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.220968 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.220988 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.220996 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb00ef0-8914-4629-a6fe-bf7eee0b3688" containerName="istio-proxy" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221018 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221026 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221038 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8e1e89ab-2973-4f78-ab89-942ac86351b0" containerName="revoke-keys" Nov 03 20:57:43.221066 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221046 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1e89ab-2973-4f78-ab89-942ac86351b0" containerName="revoke-keys" Nov 03 20:57:43.221313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221071 5815 cpu_manager.go:414] "containerMap: removing container" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" Nov 03 20:57:43.221313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221078 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" Nov 03 20:57:43.221313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221107 5815 cpu_manager.go:414] "containerMap: removing container" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" Nov 03 20:57:43.221313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221114 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" Nov 03 20:57:43.221313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221160 5815 cpu_manager.go:414] "containerMap: removing container" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" Nov 03 20:57:43.221313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221168 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346a0e2d-1843-4125-bc93-b6bf25993c3b" containerName="istio-proxy" Nov 03 20:57:43.221880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221840 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" Nov 03 20:57:43.221979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221886 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerName="istio-proxy" Nov 03 20:57:43.221979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221897 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1e89ab-2973-4f78-ab89-942ac86351b0" containerName="revoke-keys" Nov 03 20:57:43.221979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221914 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" Nov 03 20:57:43.221979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.221928 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="540934a6-2005-4927-b558-441ba575a73d" containerName="istio-proxy" Nov 03 20:57:43.241274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.241249 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5"] Nov 03 20:57:43.241458 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.241393 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.249457 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.249420 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"kube-root-ca.crt\"" Nov 03 20:57:43.249606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.249471 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"openshift-service-ca.crt\"" Nov 03 20:57:43.249606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.249557 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"sub-test-model-bab2c3-kserve-self-signed-certs\"" Nov 03 20:57:43.250041 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.250025 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bab2c3\"/\"default-dockercfg-vvf7p\"" Nov 03 20:57:43.324689 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.324651 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.324863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.324710 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsj7\" (UniqueName: \"kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.324986 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.324966 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.325028 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.325017 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.325164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.325139 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.325226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.325200 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.431529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.430555 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.431529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.430652 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.431529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.430722 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.431529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.430765 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.431529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.430823 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.431529 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.430856 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsj7\" (UniqueName: \"kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.432036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.431770 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.432036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.432024 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.434557 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.432231 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.442141 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.440712 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.478657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.472357 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsj7\" (UniqueName: \"kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.478657 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.472904 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm\") pod \"sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.546543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.543285 5815 generic.go:356] "Generic (PLEG): container finished" podID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" containerID="65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215" exitCode=0 Nov 03 20:57:43.546543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.543526 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" event={"ID":"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0","Type":"ContainerDied","Data":"65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215"} Nov 03 20:57:43.546543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.543582 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" event={"ID":"2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0","Type":"ContainerDied","Data":"9808decf9ca928cb1ee666275f2278b1732489d9df5decffd2cbfab876c65aa5"} Nov 03 20:57:43.546543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.543613 5815 scope.go:122] "RemoveContainer" containerID="65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215" Nov 03 20:57:43.546543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.543803 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2" Nov 03 20:57:43.559961 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.559934 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:43.648839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.630837 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2"] Nov 03 20:57:43.648839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.635767 5815 scope.go:122] "RemoveContainer" containerID="a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246" Nov 03 20:57:43.648839 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.638370 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-twsx2"] Nov 03 20:57:43.669044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.668774 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd"] Nov 03 20:57:43.669044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.669031 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" containerID="cri-o://9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f" gracePeriod=30 Nov 03 20:57:43.672406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.672377 5815 scope.go:122] "RemoveContainer" containerID="65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215" Nov 03 20:57:43.674013 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:43.673956 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215\": container with ID starting with 65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215 not found: ID does not exist" containerID="65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215" Nov 03 20:57:43.674144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.674002 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215"} err="failed to get container status \"65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215\": rpc error: code = NotFound desc = could not find container \"65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215\": container with ID starting with 65ee430c73f7e3c7dfddffdd9351636c1c312c9f8200d95899457b1d79aab215 not found: ID does not exist" Nov 03 20:57:43.674144 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.674032 5815 scope.go:122] "RemoveContainer" containerID="a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246" Nov 03 20:57:43.675075 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:43.675045 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246\": container with ID starting with a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246 not found: ID does not exist" containerID="a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246" Nov 03 20:57:43.675242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.675220 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246"} err="failed to get container status \"a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246\": rpc error: code = NotFound desc = could not find container \"a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246\": container with ID starting with a448153d01929fe4e12649602a1c5f46667db250b01a74dc39ab9a4faaf45246 not found: ID does not exist" Nov 03 20:57:43.857216 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:43.857184 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:57:43.951916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:43.951891 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5"] Nov 03 20:57:43.952627 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:57:43.952600 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d29238_aee6_4218_9234_af485c8df7c2.slice/crio-e75dfd42004b6d82b585b8ff8393fb7132364f881526aa6d402ab1d2855edd50 WatchSource:0}: Error finding container e75dfd42004b6d82b585b8ff8393fb7132364f881526aa6d402ab1d2855edd50: Status 404 returned error can't find the container with id e75dfd42004b6d82b585b8ff8393fb7132364f881526aa6d402ab1d2855edd50 Nov 03 20:57:44.556050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:44.556014 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" event={"ID":"61d29238-aee6-4218-9234-af485c8df7c2","Type":"ContainerStarted","Data":"a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab"} Nov 03 20:57:44.556250 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:44.556061 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" event={"ID":"61d29238-aee6-4218-9234-af485c8df7c2","Type":"ContainerStarted","Data":"e75dfd42004b6d82b585b8ff8393fb7132364f881526aa6d402ab1d2855edd50"} Nov 03 20:57:44.560710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:44.560689 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:44.581117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:44.581072 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" podStartSLOduration=1.581057574 podStartE2EDuration="1.581057574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:44.575490099 +0000 UTC m=+3374.105835773" watchObservedRunningTime="2026-11-03 20:57:44.581057574 +0000 UTC m=+3374.111403250" Nov 03 20:57:44.589779 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:44.589746 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0" path="/var/lib/kubelet/pods/2b5ecf4d-7071-4b55-a6e2-de1108ce8ff0/volumes" Nov 03 20:57:46.385976 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:46.385939 5815 scope.go:122] "RemoveContainer" containerID="d0fd1d7f0d7d0894d14f9588a092d44cdc39d2186d86740e2a637e3634a828df" Nov 03 20:57:46.406532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:46.405921 5815 scope.go:122] "RemoveContainer" containerID="1cc4df764316191e85cc7e0ff0e8b935c28c2fbbdc18fc23af57b7cb85ba33f7" Nov 03 20:57:46.417126 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:46.417105 5815 scope.go:122] "RemoveContainer" containerID="32e4be165d68959d264015bf86e8da3f583869bcfa8d6bfd7286142567ba33b3" Nov 03 20:57:46.427975 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:46.427957 5815 scope.go:122] "RemoveContainer" containerID="6e17f8ba8b2044c8281508d33e37e302f4f329542d6c904779b67db619b4f566" Nov 03 20:57:48.957683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:48.957617 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:57:49.031861 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.031821 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.032052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.031916 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.032052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.031981 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.032052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032030 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.032389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032211 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:49.032389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032220 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.032389 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032346 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032423 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032531 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032545 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032618 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket\") pod \"79dae19a-562f-4e3e-9136-81672c805be1\" (UID: \"79dae19a-562f-4e3e-9136-81672c805be1\") " Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032743 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.032919 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:49.033472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.033093 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:49.034326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.033472 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.034326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.033499 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79dae19a-562f-4e3e-9136-81672c805be1-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.034326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.033538 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.034326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.033554 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.034326 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.033571 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.036168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.036138 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:49.036401 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.036375 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:49.037602 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.037074 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "kube-api-access-t2xmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:49.038036 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.038006 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token" pod "79dae19a-562f-4e3e-9136-81672c805be1" (UID: "79dae19a-562f-4e3e-9136-81672c805be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:49.135125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.135091 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79dae19a-562f-4e3e-9136-81672c805be1-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.135125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.135120 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2xmh\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-kube-api-access-t2xmh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.135125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.135131 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79dae19a-562f-4e3e-9136-81672c805be1-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.135391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.135140 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79dae19a-562f-4e3e-9136-81672c805be1-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.599413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.599378 5815 generic.go:356] "Generic (PLEG): container finished" podID="79dae19a-562f-4e3e-9136-81672c805be1" containerID="9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f" exitCode=0 Nov 03 20:57:49.599646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.599423 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" Nov 03 20:57:49.599646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.599442 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" event={"ID":"79dae19a-562f-4e3e-9136-81672c805be1","Type":"ContainerDied","Data":"9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f"} Nov 03 20:57:49.599646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.599521 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd" event={"ID":"79dae19a-562f-4e3e-9136-81672c805be1","Type":"ContainerDied","Data":"7d5b1ddc6147297f7430446f8149b55b3dd8ce60a8e9b6ea4c135993a3cf6d9f"} Nov 03 20:57:49.599646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.599555 5815 scope.go:122] "RemoveContainer" containerID="9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f" Nov 03 20:57:49.618806 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.618782 5815 scope.go:122] "RemoveContainer" containerID="9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f" Nov 03 20:57:49.619547 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:49.619459 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f\": container with ID starting with 9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f not found: ID does not exist" containerID="9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f" Nov 03 20:57:49.619664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.619559 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f"} err="failed to get container status \"9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f\": rpc error: code = NotFound desc = could not find container \"9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f\": container with ID starting with 9378eee798d5e6b4614a43840bd33c3eb11b9b46f4bc79ac5debaba19e55aa9f not found: ID does not exist" Nov 03 20:57:49.624125 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.624101 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74"] Nov 03 20:57:49.624385 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.624339 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" containerID="cri-o://e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a" gracePeriod=30 Nov 03 20:57:49.638328 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.638303 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd"] Nov 03 20:57:49.650592 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:49.650568 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-8v2qd"] Nov 03 20:57:50.594962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:50.594924 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79dae19a-562f-4e3e-9136-81672c805be1" path="/var/lib/kubelet/pods/79dae19a-562f-4e3e-9136-81672c805be1/volumes" Nov 03 20:57:53.560546 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:53.560492 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:53.573254 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:53.573227 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:53.668820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:53.668791 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 20:57:54.657825 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:54.657332 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv"] Nov 03 20:57:55.160350 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.159922 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:57:55.186681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.186637 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc"] Nov 03 20:57:55.187337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.187321 5815 cpu_manager.go:414] "containerMap: removing container" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" Nov 03 20:57:55.187337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.187336 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" Nov 03 20:57:55.187477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.187369 5815 cpu_manager.go:414] "containerMap: removing container" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" Nov 03 20:57:55.187477 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.187374 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" Nov 03 20:57:55.187664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.187490 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="79dae19a-562f-4e3e-9136-81672c805be1" containerName="istio-proxy" Nov 03 20:57:55.187664 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.187522 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerName="istio-proxy" Nov 03 20:57:55.191403 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.191371 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.196675 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.196628 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:57:55.198033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.197353 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:55.230033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.229943 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc"] Nov 03 20:57:55.301942 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.301905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.301987 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302014 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302114 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302153 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302136 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302219 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:55.302406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302242 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302377 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:55.302693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302402 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302440 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302488 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:55.302693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302407 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:55.302693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302538 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy\") pod \"9fd93595-2633-406e-9d75-5fcf5b1cc835\" (UID: \"9fd93595-2633-406e-9d75-5fcf5b1cc835\") " Nov 03 20:57:55.302693 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302660 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:55.303043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.302828 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p48s\" (UniqueName: \"kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.303111 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303057 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.303495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303278 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.303495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303428 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.303495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303440 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd93595-2633-406e-9d75-5fcf5b1cc835-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.303495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303450 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.303495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303465 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.303495 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.303474 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.304796 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.304765 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:55.305359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.305336 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:55.305359 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.305342 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:55.306031 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.306015 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv" pod "9fd93595-2633-406e-9d75-5fcf5b1cc835" (UID: "9fd93595-2633-406e-9d75-5fcf5b1cc835"). InnerVolumeSpecName "kube-api-access-lbkpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:55.404951 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.404910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.405171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.404997 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p48s\" (UniqueName: \"kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.405171 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.405161 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.405423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.405383 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.405423 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.405410 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.405573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.405425 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd93595-2633-406e-9d75-5fcf5b1cc835-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.405573 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.405455 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbkpv\" (UniqueName: \"kubernetes.io/projected/9fd93595-2633-406e-9d75-5fcf5b1cc835-kube-api-access-lbkpv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.405769 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.405743 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.407955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.407932 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.417347 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.417321 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p48s\" (UniqueName: \"kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s\") pod \"maas-api-revoke-sub-7b2b7183-8ptnc\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.520695 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.520620 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:55.696050 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.696018 5815 generic.go:356] "Generic (PLEG): container finished" podID="9fd93595-2633-406e-9d75-5fcf5b1cc835" containerID="e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a" exitCode=0 Nov 03 20:57:55.696482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.696067 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" Nov 03 20:57:55.696482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.696099 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" event={"ID":"9fd93595-2633-406e-9d75-5fcf5b1cc835","Type":"ContainerDied","Data":"e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a"} Nov 03 20:57:55.696482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.696158 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74" event={"ID":"9fd93595-2633-406e-9d75-5fcf5b1cc835","Type":"ContainerDied","Data":"7a3193d60d317a4434e3d1a833c2423d2ff1f8fe05283565e11b0ec1b281bb8e"} Nov 03 20:57:55.696482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.696186 5815 scope.go:122] "RemoveContainer" containerID="e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a" Nov 03 20:57:55.709708 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.709687 5815 scope.go:122] "RemoveContainer" containerID="e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a" Nov 03 20:57:55.710003 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:57:55.709978 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a\": container with ID starting with e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a not found: ID does not exist" containerID="e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a" Nov 03 20:57:55.710100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.710014 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a"} err="failed to get container status \"e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a\": rpc error: code = NotFound desc = could not find container \"e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a\": container with ID starting with e60bb6f9d426c0cc8ca2763f8b9a12bb906bec815daa9ba52d5077498f0d779a not found: ID does not exist" Nov 03 20:57:55.726960 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.726938 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74"] Nov 03 20:57:55.731743 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.731725 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-88w74"] Nov 03 20:57:55.808392 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:57:55.808359 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb74f0de_2ad0_4c18_bd64_872e26ec682a.slice/crio-a3f61f1b626f08a3b69844d4b793b79560230f65e3d50f530cb1c8f838efaae9 WatchSource:0}: Error finding container a3f61f1b626f08a3b69844d4b793b79560230f65e3d50f530cb1c8f838efaae9: Status 404 returned error can't find the container with id a3f61f1b626f08a3b69844d4b793b79560230f65e3d50f530cb1c8f838efaae9 Nov 03 20:57:55.809846 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:55.809821 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc"] Nov 03 20:57:56.590745 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:56.590710 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd93595-2633-406e-9d75-5fcf5b1cc835" path="/var/lib/kubelet/pods/9fd93595-2633-406e-9d75-5fcf5b1cc835/volumes" Nov 03 20:57:56.711294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:56.711261 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" event={"ID":"db74f0de-2ad0-4c18-bd64-872e26ec682a","Type":"ContainerStarted","Data":"f385928f5763b8c2870d02595f157af935f62884451cc9485d22d4d21b562657"} Nov 03 20:57:56.711842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:56.711326 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" event={"ID":"db74f0de-2ad0-4c18-bd64-872e26ec682a","Type":"ContainerStarted","Data":"a3f61f1b626f08a3b69844d4b793b79560230f65e3d50f530cb1c8f838efaae9"} Nov 03 20:57:56.733555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:56.733496 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" podStartSLOduration=1.7334838129999999 podStartE2EDuration="1.733483813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:56.72920005 +0000 UTC m=+3386.259545723" watchObservedRunningTime="2026-11-03 20:57:56.733483813 +0000 UTC m=+3386.263829485" Nov 03 20:57:57.724904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:57.724870 5815 generic.go:356] "Generic (PLEG): container finished" podID="db74f0de-2ad0-4c18-bd64-872e26ec682a" containerID="f385928f5763b8c2870d02595f157af935f62884451cc9485d22d4d21b562657" exitCode=0 Nov 03 20:57:57.725391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:57.724955 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" event={"ID":"db74f0de-2ad0-4c18-bd64-872e26ec682a","Type":"ContainerDied","Data":"f385928f5763b8c2870d02595f157af935f62884451cc9485d22d4d21b562657"} Nov 03 20:57:58.932115 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:58.932091 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:57:59.083478 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.083393 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca\") pod \"db74f0de-2ad0-4c18-bd64-872e26ec682a\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " Nov 03 20:57:59.083916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.083487 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca\") pod \"db74f0de-2ad0-4c18-bd64-872e26ec682a\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " Nov 03 20:57:59.083916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.083803 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca" pod "db74f0de-2ad0-4c18-bd64-872e26ec682a" (UID: "db74f0de-2ad0-4c18-bd64-872e26ec682a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:59.083916 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.083840 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s\" (UniqueName: \"kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s\") pod \"db74f0de-2ad0-4c18-bd64-872e26ec682a\" (UID: \"db74f0de-2ad0-4c18-bd64-872e26ec682a\") " Nov 03 20:57:59.084688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.084646 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:59.086042 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.086015 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca" pod "db74f0de-2ad0-4c18-bd64-872e26ec682a" (UID: "db74f0de-2ad0-4c18-bd64-872e26ec682a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:59.086955 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.086922 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s" pod "db74f0de-2ad0-4c18-bd64-872e26ec682a" (UID: "db74f0de-2ad0-4c18-bd64-872e26ec682a"). InnerVolumeSpecName "kube-api-access-9p48s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:59.186037 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.186006 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p48s\" (UniqueName: \"kubernetes.io/projected/db74f0de-2ad0-4c18-bd64-872e26ec682a-kube-api-access-9p48s\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:59.186211 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.186041 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db74f0de-2ad0-4c18-bd64-872e26ec682a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:59.754413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.754369 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" event={"ID":"db74f0de-2ad0-4c18-bd64-872e26ec682a","Type":"ContainerDied","Data":"a3f61f1b626f08a3b69844d4b793b79560230f65e3d50f530cb1c8f838efaae9"} Nov 03 20:57:59.754634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.754478 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f61f1b626f08a3b69844d4b793b79560230f65e3d50f530cb1c8f838efaae9" Nov 03 20:57:59.754634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:57:59.754595 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc" Nov 03 20:58:01.222137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:01.222104 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk"] Nov 03 20:58:04.212962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.210969 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj"] Nov 03 20:58:04.212962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.212057 5815 cpu_manager.go:414] "containerMap: removing container" podUID="db74f0de-2ad0-4c18-bd64-872e26ec682a" containerName="revoke-keys" Nov 03 20:58:04.212962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.212078 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db74f0de-2ad0-4c18-bd64-872e26ec682a" containerName="revoke-keys" Nov 03 20:58:04.212962 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.212266 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="db74f0de-2ad0-4c18-bd64-872e26ec682a" containerName="revoke-keys" Nov 03 20:58:04.221979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.221639 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj"] Nov 03 20:58:04.221979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.221741 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.245023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.242350 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"default-dockercfg-jhptg\"" Nov 03 20:58:04.245023 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.243766 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"openshift-service-ca.crt\"" Nov 03 20:58:04.250092 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.249690 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"kube-root-ca.crt\"" Nov 03 20:58:04.286609 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.286526 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7454\" (UniqueName: \"kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.286770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.286633 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.286770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.286684 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.286770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.286746 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.286965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.286881 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.286965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.286917 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.332893 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.332868 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6e5139\"/\"sub-test-model-6e5139-kserve-self-signed-certs\"" Nov 03 20:58:04.388137 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388102 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7454\" (UniqueName: \"kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388155 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388188 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388413 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388688 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388665 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388819 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388717 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388888 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.388946 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.388668 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.389259 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.389202 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.391187 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.391099 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.392386 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.392361 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.408416 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.407407 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7454\" (UniqueName: \"kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454\") pod \"sub-test-model-6e5139-kserve-5894d74b59-wzvmj\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.562936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.562819 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:04.908325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:04.908294 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj"] Nov 03 20:58:04.908471 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:58:04.908452 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f803152_09a3_4007_8a43_612b71e83a63.slice/crio-503b82729d488b1152681706d56b05be8a4a3233a22924e91639b4bb1b97582c WatchSource:0}: Error finding container 503b82729d488b1152681706d56b05be8a4a3233a22924e91639b4bb1b97582c: Status 404 returned error can't find the container with id 503b82729d488b1152681706d56b05be8a4a3233a22924e91639b4bb1b97582c Nov 03 20:58:05.819013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:05.818981 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" event={"ID":"1f803152-09a3-4007-8a43-612b71e83a63","Type":"ContainerStarted","Data":"36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7"} Nov 03 20:58:05.819013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:05.819023 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" event={"ID":"1f803152-09a3-4007-8a43-612b71e83a63","Type":"ContainerStarted","Data":"503b82729d488b1152681706d56b05be8a4a3233a22924e91639b4bb1b97582c"} Nov 03 20:58:05.839320 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:05.839274 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" podStartSLOduration=1.839260478 podStartE2EDuration="1.839260478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:05.837899372 +0000 UTC m=+3395.368245047" watchObservedRunningTime="2026-11-03 20:58:05.839260478 +0000 UTC m=+3395.369606174" Nov 03 20:58:06.223411 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:06.223371 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerName="payload-pre-processing" containerID="cri-o://966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6" gracePeriod=25 Nov 03 20:58:06.563621 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:06.563543 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:08.778667 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:08.778632 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:58:08.937531 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:08.936602 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:58:09.707755 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:09.707714 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" containerID="cri-o://acfabede5d0a0be91799c606734cfedb9578d6aa3448307aa04538b94bac82a2" gracePeriod=45 Nov 03 20:58:09.763768 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:09.763741 5815 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5fa09b_73f7_445d_9c96_74a8227fd3b5.slice/crio-acfabede5d0a0be91799c606734cfedb9578d6aa3448307aa04538b94bac82a2.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:58:09.904392 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:09.904355 5815 generic.go:356] "Generic (PLEG): container finished" podID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerID="acfabede5d0a0be91799c606734cfedb9578d6aa3448307aa04538b94bac82a2" exitCode=0 Nov 03 20:58:09.904936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:09.904644 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" event={"ID":"1e5fa09b-73f7-445d-9c96-74a8227fd3b5","Type":"ContainerDied","Data":"acfabede5d0a0be91799c606734cfedb9578d6aa3448307aa04538b94bac82a2"} Nov 03 20:58:10.135533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.134975 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:58:10.245043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245012 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir\") pod \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " Nov 03 20:58:10.245352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245335 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x\" (UniqueName: \"kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x\") pod \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " Nov 03 20:58:10.245474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245460 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs\") pod \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " Nov 03 20:58:10.245659 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245646 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm\") pod \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " Nov 03 20:58:10.245799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245785 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache\") pod \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " Nov 03 20:58:10.246003 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245916 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home\") pod \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\" (UID: \"1e5fa09b-73f7-445d-9c96-74a8227fd3b5\") " Nov 03 20:58:10.246166 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.245670 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir" pod "1e5fa09b-73f7-445d-9c96-74a8227fd3b5" (UID: "1e5fa09b-73f7-445d-9c96-74a8227fd3b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:10.247034 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.247008 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:10.247264 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.247224 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home" pod "1e5fa09b-73f7-445d-9c96-74a8227fd3b5" (UID: "1e5fa09b-73f7-445d-9c96-74a8227fd3b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:10.248858 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.248836 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache" pod "1e5fa09b-73f7-445d-9c96-74a8227fd3b5" (UID: "1e5fa09b-73f7-445d-9c96-74a8227fd3b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:10.249965 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.249928 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs" pod "1e5fa09b-73f7-445d-9c96-74a8227fd3b5" (UID: "1e5fa09b-73f7-445d-9c96-74a8227fd3b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:10.251754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.251720 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm" pod "1e5fa09b-73f7-445d-9c96-74a8227fd3b5" (UID: "1e5fa09b-73f7-445d-9c96-74a8227fd3b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:10.253052 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.252984 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x" pod "1e5fa09b-73f7-445d-9c96-74a8227fd3b5" (UID: "1e5fa09b-73f7-445d-9c96-74a8227fd3b5"). InnerVolumeSpecName "kube-api-access-srl2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:10.304761 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.304731 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5"] Nov 03 20:58:10.349973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.349850 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:10.349973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.349883 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srl2x\" (UniqueName: \"kubernetes.io/projected/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-kube-api-access-srl2x\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:10.349973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.349902 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:10.349973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.349919 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:10.349973 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.349934 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e5fa09b-73f7-445d-9c96-74a8227fd3b5-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:10.916224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.916149 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" Nov 03 20:58:10.916224 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.916179 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv" event={"ID":"1e5fa09b-73f7-445d-9c96-74a8227fd3b5","Type":"ContainerDied","Data":"ea1531c4ac76f9e4a21e11d56f7644c46eaf21c4215f3a139e098ecbd56e7727"} Nov 03 20:58:10.916772 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.916232 5815 scope.go:122] "RemoveContainer" containerID="acfabede5d0a0be91799c606734cfedb9578d6aa3448307aa04538b94bac82a2" Nov 03 20:58:10.944770 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.944746 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv"] Nov 03 20:58:10.952465 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:10.952438 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-745c52/test-model-745c52-kserve-696c945f87-kxnbv"] Nov 03 20:58:11.025881 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:11.025849 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g"] Nov 03 20:58:11.031543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:11.031499 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc47b212-vmh8g"] Nov 03 20:58:12.590322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:12.590285 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" path="/var/lib/kubelet/pods/1e5fa09b-73f7-445d-9c96-74a8227fd3b5/volumes" Nov 03 20:58:12.591043 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:12.591019 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4de4f0c-19aa-4b2b-af0d-839417c6db09" path="/var/lib/kubelet/pods/d4de4f0c-19aa-4b2b-af0d-839417c6db09/volumes" Nov 03 20:58:13.853988 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:13.853944 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:58:14.563938 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:14.563905 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:14.576798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:14.576768 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:14.974880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:14.974838 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 20:58:18.879078 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:18.879032 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:58:23.683876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.683838 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6"] Nov 03 20:58:23.684614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.684593 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" Nov 03 20:58:23.684614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.684610 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" Nov 03 20:58:23.684811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.684757 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5fa09b-73f7-445d-9c96-74a8227fd3b5" containerName="main" Nov 03 20:58:23.689767 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.689747 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:23.691807 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.691777 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47"] Nov 03 20:58:23.697201 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.697174 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:23.704267 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.704224 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3bcc2b\"" Nov 03 20:58:23.705378 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.705354 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-3bcc2b-dockercfg-ksvzv\"" Nov 03 20:58:23.715618 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.715588 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6"] Nov 03 20:58:23.871931 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.871895 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47"] Nov 03 20:58:23.986597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.962184 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb9q\" (UniqueName: \"kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q\") pod \"payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:23.986597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.962259 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrnb\" (UniqueName: \"kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb\") pod \"payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:23.986597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.962318 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:23.986597 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:23.962810 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:24.064106 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.064062 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrnb\" (UniqueName: \"kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb\") pod \"payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:24.064339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.064144 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:24.064339 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.064295 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:24.064555 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.064487 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb9q\" (UniqueName: \"kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q\") pod \"payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:24.065091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.065067 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:24.065189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.065164 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:24.077000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.076966 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb9q\" (UniqueName: \"kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q\") pod \"payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:24.077193 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.077172 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrnb\" (UniqueName: \"kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb\") pod \"payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:24.307186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.307099 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:24.319751 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.319723 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:24.597812 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:58:24.597771 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc693e7b8_e44b_4fd8_843e_0a686c5fcce2.slice/crio-015665e8d2dcafb79fe3f1206201067482779041f937f1cfbf35ee3149f07210 WatchSource:0}: Error finding container 015665e8d2dcafb79fe3f1206201067482779041f937f1cfbf35ee3149f07210: Status 404 returned error can't find the container with id 015665e8d2dcafb79fe3f1206201067482779041f937f1cfbf35ee3149f07210 Nov 03 20:58:24.599883 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.599854 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6"] Nov 03 20:58:24.630903 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:24.630881 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47"] Nov 03 20:58:24.631089 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:58:24.631064 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572f9e0c_4664_4a36_a52f_7936058aa1e4.slice/crio-516b43d481f8683564c432f0a350f3a90328a0c5d5f2314adf00a2e1c112afd9 WatchSource:0}: Error finding container 516b43d481f8683564c432f0a350f3a90328a0c5d5f2314adf00a2e1c112afd9: Status 404 returned error can't find the container with id 516b43d481f8683564c432f0a350f3a90328a0c5d5f2314adf00a2e1c112afd9 Nov 03 20:58:25.068681 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.068645 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" event={"ID":"572f9e0c-4664-4a36-a52f-7936058aa1e4","Type":"ContainerStarted","Data":"631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21"} Nov 03 20:58:25.069151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.068763 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" event={"ID":"572f9e0c-4664-4a36-a52f-7936058aa1e4","Type":"ContainerStarted","Data":"516b43d481f8683564c432f0a350f3a90328a0c5d5f2314adf00a2e1c112afd9"} Nov 03 20:58:25.069151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.068799 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:25.071437 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.071407 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" event={"ID":"c693e7b8-e44b-4fd8-843e-0a686c5fcce2","Type":"ContainerStarted","Data":"2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122"} Nov 03 20:58:25.071648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.071460 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" event={"ID":"c693e7b8-e44b-4fd8-843e-0a686c5fcce2","Type":"ContainerStarted","Data":"015665e8d2dcafb79fe3f1206201067482779041f937f1cfbf35ee3149f07210"} Nov 03 20:58:25.071648 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.071492 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:25.089660 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.089618 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" podStartSLOduration=2.089605772 podStartE2EDuration="2.089605772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:25.087773266 +0000 UTC m=+3414.618118939" watchObservedRunningTime="2026-11-03 20:58:25.089605772 +0000 UTC m=+3414.619951444" Nov 03 20:58:25.112810 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.112761 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" podStartSLOduration=2.112749138 podStartE2EDuration="2.112749138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:25.109174612 +0000 UTC m=+3414.639520284" watchObservedRunningTime="2026-11-03 20:58:25.112749138 +0000 UTC m=+3414.643094809" Nov 03 20:58:25.349000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.348904 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" containerID="cri-o://82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f" gracePeriod=45 Nov 03 20:58:25.680222 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.680191 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:58:25.792177 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792144 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs\" (UniqueName: \"kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs\") pod \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " Nov 03 20:58:25.792396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792221 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5\" (UniqueName: \"kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5\") pod \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " Nov 03 20:58:25.792396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792256 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home\") pod \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " Nov 03 20:58:25.792396 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792336 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm\") pod \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " Nov 03 20:58:25.792638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792463 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir\") pod \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " Nov 03 20:58:25.792638 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792609 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache\") pod \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\" (UID: \"77ae31d3-cee4-44d9-a25c-381f9f1a5472\") " Nov 03 20:58:25.792927 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792899 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir" pod "77ae31d3-cee4-44d9-a25c-381f9f1a5472" (UID: "77ae31d3-cee4-44d9-a25c-381f9f1a5472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:25.793234 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.792930 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache" pod "77ae31d3-cee4-44d9-a25c-381f9f1a5472" (UID: "77ae31d3-cee4-44d9-a25c-381f9f1a5472"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:25.793429 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.793408 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:25.793543 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.793445 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:25.794430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.794403 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home" pod "77ae31d3-cee4-44d9-a25c-381f9f1a5472" (UID: "77ae31d3-cee4-44d9-a25c-381f9f1a5472"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:25.795197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.795164 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs" pod "77ae31d3-cee4-44d9-a25c-381f9f1a5472" (UID: "77ae31d3-cee4-44d9-a25c-381f9f1a5472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:25.795487 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.795470 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5" pod "77ae31d3-cee4-44d9-a25c-381f9f1a5472" (UID: "77ae31d3-cee4-44d9-a25c-381f9f1a5472"). InnerVolumeSpecName "kube-api-access-hg6h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:25.796040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.796016 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm" pod "77ae31d3-cee4-44d9-a25c-381f9f1a5472" (UID: "77ae31d3-cee4-44d9-a25c-381f9f1a5472"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:25.894683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.894602 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ae31d3-cee4-44d9-a25c-381f9f1a5472-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:25.894683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.894633 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6h5\" (UniqueName: \"kubernetes.io/projected/77ae31d3-cee4-44d9-a25c-381f9f1a5472-kube-api-access-hg6h5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:25.894683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.894643 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:25.894683 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:25.894651 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ae31d3-cee4-44d9-a25c-381f9f1a5472-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:26.083677 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.083641 5815 generic.go:356] "Generic (PLEG): container finished" podID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerID="82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f" exitCode=0 Nov 03 20:58:26.084179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.083686 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" Nov 03 20:58:26.084179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.083722 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" event={"ID":"77ae31d3-cee4-44d9-a25c-381f9f1a5472","Type":"ContainerDied","Data":"82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f"} Nov 03 20:58:26.084179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.083776 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5" event={"ID":"77ae31d3-cee4-44d9-a25c-381f9f1a5472","Type":"ContainerDied","Data":"7c46de3afb489d81941cbd9e32951f99a1fcdb8bf794647ad9362340159a7593"} Nov 03 20:58:26.084179 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.083799 5815 scope.go:122] "RemoveContainer" containerID="82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f" Nov 03 20:58:26.126100 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.126065 5815 scope.go:122] "RemoveContainer" containerID="82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f" Nov 03 20:58:26.128722 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:26.128651 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f\": container with ID starting with 82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f not found: ID does not exist" containerID="82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f" Nov 03 20:58:26.128722 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.128696 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f"} err="failed to get container status \"82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f\": rpc error: code = NotFound desc = could not find container \"82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f\": container with ID starting with 82619678403c3579b0f57618bafda445ecdacc40f818cc0f889a58cbe3cfba1f not found: ID does not exist" Nov 03 20:58:26.140279 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.140253 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5"] Nov 03 20:58:26.145266 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.145215 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5ecee8/test-model-5ecee8-kserve-699b54d677-7k7q5"] Nov 03 20:58:26.546654 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.546630 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:58:26.614365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.614329 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" path="/var/lib/kubelet/pods/77ae31d3-cee4-44d9-a25c-381f9f1a5472/volumes" Nov 03 20:58:26.618322 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.618294 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume\") pod \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " Nov 03 20:58:26.618600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.618579 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx\") pod \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\" (UID: \"ee02d163-eb9c-40e4-89a9-687da6fde6bf\") " Nov 03 20:58:26.619535 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.619488 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume" pod "ee02d163-eb9c-40e4-89a9-687da6fde6bf" (UID: "ee02d163-eb9c-40e4-89a9-687da6fde6bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:26.626000 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.625975 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx" pod "ee02d163-eb9c-40e4-89a9-687da6fde6bf" (UID: "ee02d163-eb9c-40e4-89a9-687da6fde6bf"). InnerVolumeSpecName "kube-api-access-szrhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:26.722542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.721997 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee02d163-eb9c-40e4-89a9-687da6fde6bf-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:26.722542 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.722036 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/ee02d163-eb9c-40e4-89a9-687da6fde6bf-kube-api-access-szrhx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:26.974292 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.974196 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh"] Nov 03 20:58:26.974547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:26.974494 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" containerID="cri-o://2631a874f01034ff7ac630971e18a0f77db9d1731551ed7bde4fb5c73e3f12f4" gracePeriod=30 Nov 03 20:58:27.175192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.175154 5815 generic.go:356] "Generic (PLEG): container finished" podID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerID="966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6" exitCode=0 Nov 03 20:58:27.176556 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.176016 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" Nov 03 20:58:27.180088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.176039 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" event={"ID":"ee02d163-eb9c-40e4-89a9-687da6fde6bf","Type":"ContainerDied","Data":"966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6"} Nov 03 20:58:27.180742 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.180718 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk" event={"ID":"ee02d163-eb9c-40e4-89a9-687da6fde6bf","Type":"ContainerDied","Data":"4fca6da67878911f52b94ae852977f2c7210a9f959651dd8e1813a25f5d27d6e"} Nov 03 20:58:27.180968 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.180892 5815 scope.go:122] "RemoveContainer" containerID="966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6" Nov 03 20:58:27.243443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.243120 5815 scope.go:122] "RemoveContainer" containerID="966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6" Nov 03 20:58:27.243822 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:27.243722 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6\": container with ID starting with 966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6 not found: ID does not exist" containerID="966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6" Nov 03 20:58:27.243822 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.243760 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6"} err="failed to get container status \"966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6\": rpc error: code = NotFound desc = could not find container \"966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6\": container with ID starting with 966a82e17ab922e53b3743887b896ec8ae78936e2d206bbafb6b6fcdd3d87db6 not found: ID does not exist" Nov 03 20:58:27.277430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.277395 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk"] Nov 03 20:58:27.302226 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:27.301107 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8d0fa-c45cbd755-tsgmk"] Nov 03 20:58:28.600801 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:28.600725 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" path="/var/lib/kubelet/pods/ee02d163-eb9c-40e4-89a9-687da6fde6bf/volumes" Nov 03 20:58:29.025175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:29.025144 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j"] Nov 03 20:58:29.029295 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:29.029268 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4857b0b9-dwv4j"] Nov 03 20:58:29.565603 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:29.565566 5815 patch_prober.go:29] interesting pod/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:29.565787 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:29.565636 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:30.595253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:30.595212 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27389859-8742-44a6-b6c9-372928569080" path="/var/lib/kubelet/pods/27389859-8742-44a6-b6c9-372928569080/volumes" Nov 03 20:58:32.233949 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.233917 5815 generic.go:356] "Generic (PLEG): container finished" podID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerID="2631a874f01034ff7ac630971e18a0f77db9d1731551ed7bde4fb5c73e3f12f4" exitCode=0 Nov 03 20:58:32.234474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.234020 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" event={"ID":"0e375507-e16f-4bb4-8187-39e26e4e5348","Type":"ContainerDied","Data":"2631a874f01034ff7ac630971e18a0f77db9d1731551ed7bde4fb5c73e3f12f4"} Nov 03 20:58:32.234474 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.234081 5815 scope.go:122] "RemoveContainer" containerID="8a95ae10a92e560c4b9b641179e4f692f55b4cf88a902cc446a36dd28c78aa23" Nov 03 20:58:32.320527 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.320477 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:58:32.347876 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.347844 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.348057 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.347964 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.348196 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.348113 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.348274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.348147 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.348274 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.348230 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.348813 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.348492 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:32.348920 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.348834 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:32.349091 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.349070 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:32.349204 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.349173 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.349330 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.349314 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.349413 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.349387 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.349496 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.349482 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket\") pod \"0e375507-e16f-4bb4-8187-39e26e4e5348\" (UID: \"0e375507-e16f-4bb4-8187-39e26e4e5348\") " Nov 03 20:58:32.350059 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.349893 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:32.350164 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.350112 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:32.350364 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.350345 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.350412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.350370 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.350412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.350385 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.350412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.350400 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e375507-e16f-4bb4-8187-39e26e4e5348-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.352377 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.350414 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.359009 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.358984 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:32.359105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.359007 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "kube-api-access-j4xfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:32.359105 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.359041 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:32.361168 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.361146 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy" pod "0e375507-e16f-4bb4-8187-39e26e4e5348" (UID: "0e375507-e16f-4bb4-8187-39e26e4e5348"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:32.451644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.451613 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.451644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.451640 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.451644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.451651 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e375507-e16f-4bb4-8187-39e26e4e5348-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.451869 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.451662 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xfl\" (UniqueName: \"kubernetes.io/projected/0e375507-e16f-4bb4-8187-39e26e4e5348-kube-api-access-j4xfl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:32.923263 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.923178 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6"] Nov 03 20:58:32.923768 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.923712 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" containerID="cri-o://a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6" gracePeriod=30 Nov 03 20:58:32.964764 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.964733 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6"] Nov 03 20:58:32.970283 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:32.970252 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb"] Nov 03 20:58:33.331952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.329949 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" event={"ID":"0e375507-e16f-4bb4-8187-39e26e4e5348","Type":"ContainerDied","Data":"24d48d24defd36a1f6b6c3fe2a0f9eea268acedee72c33ccdfed45e631c7c1ba"} Nov 03 20:58:33.331952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.330021 5815 scope.go:122] "RemoveContainer" containerID="2631a874f01034ff7ac630971e18a0f77db9d1731551ed7bde4fb5c73e3f12f4" Nov 03 20:58:33.331952 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.330149 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh" Nov 03 20:58:33.412619 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.412580 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh"] Nov 03 20:58:33.420183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.420159 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d8d0fa-openshift-default-69f67bf7c-kvvkh"] Nov 03 20:58:33.509486 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.509455 5815 patch_prober.go:29] interesting pod/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:33.509679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:33.509525 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:34.589317 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:34.589286 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" path="/var/lib/kubelet/pods/0e375507-e16f-4bb4-8187-39e26e4e5348/volumes" Nov 03 20:58:36.087443 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:36.087408 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 20:58:36.087983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:36.087496 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 20:58:37.965646 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:37.965575 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" podUID="d2f8860d-3df3-4242-b9d5-bec04b4699aa" containerName="payload-processing" containerID="cri-o://9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b" gracePeriod=25 Nov 03 20:58:37.971015 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:37.970980 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerName="payload-pre-processing" containerID="cri-o://58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752" gracePeriod=25 Nov 03 20:58:38.265898 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.265869 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:58:38.310820 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.310790 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:58:38.353285 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353250 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn\" (UniqueName: \"kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn\") pod \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " Nov 03 20:58:38.353455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353329 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353404 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume\") pod \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\" (UID: \"d2f8860d-3df3-4242-b9d5-bec04b4699aa\") " Nov 03 20:58:38.353455 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353434 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353478 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353559 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353659 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353707 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353695 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353827 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.353958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353835 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:38.353958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353871 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume" pod "d2f8860d-3df3-4242-b9d5-bec04b4699aa" (UID: "d2f8860d-3df3-4242-b9d5-bec04b4699aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:38.353958 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.353920 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.354189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354063 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:38.354189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354177 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data\") pod \"2483d612-5b6d-43e6-936d-8a131aff8ce9\" (UID: \"2483d612-5b6d-43e6-936d-8a131aff8ce9\") " Nov 03 20:58:38.354430 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354403 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:38.354564 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354543 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:38.354690 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354661 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:38.354943 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354925 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.355044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354949 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.355044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354963 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.355044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354978 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f8860d-3df3-4242-b9d5-bec04b4699aa-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.355044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.354993 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.355044 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.355007 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2483d612-5b6d-43e6-936d-8a131aff8ce9-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.356406 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.356384 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn" pod "d2f8860d-3df3-4242-b9d5-bec04b4699aa" (UID: "d2f8860d-3df3-4242-b9d5-bec04b4699aa"). InnerVolumeSpecName "kube-api-access-dgmgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:38.356636 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.356618 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:38.356907 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.356883 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:38.356983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.356945 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "kube-api-access-q8ftc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:38.358391 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.358369 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo" pod "2483d612-5b6d-43e6-936d-8a131aff8ce9" (UID: "2483d612-5b6d-43e6-936d-8a131aff8ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:38.393239 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.393209 5815 generic.go:356] "Generic (PLEG): container finished" podID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerID="a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6" exitCode=0 Nov 03 20:58:38.393365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.393264 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" Nov 03 20:58:38.393365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.393289 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" event={"ID":"2483d612-5b6d-43e6-936d-8a131aff8ce9","Type":"ContainerDied","Data":"a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6"} Nov 03 20:58:38.393365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.393341 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6" event={"ID":"2483d612-5b6d-43e6-936d-8a131aff8ce9","Type":"ContainerDied","Data":"609f77f79554258e54f26d576d49387e1cee2b7608a82a0bc552cbe9e1afe2df"} Nov 03 20:58:38.393365 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.393362 5815 scope.go:122] "RemoveContainer" containerID="a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6" Nov 03 20:58:38.400189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.395414 5815 generic.go:356] "Generic (PLEG): container finished" podID="d2f8860d-3df3-4242-b9d5-bec04b4699aa" containerID="9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b" exitCode=0 Nov 03 20:58:38.400189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.395468 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" Nov 03 20:58:38.400189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.395533 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" event={"ID":"d2f8860d-3df3-4242-b9d5-bec04b4699aa","Type":"ContainerDied","Data":"9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b"} Nov 03 20:58:38.400189 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.395580 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6" event={"ID":"d2f8860d-3df3-4242-b9d5-bec04b4699aa","Type":"ContainerDied","Data":"431687b25a8024b263ae83f610cf1ed415be74b0db7274693de2f8ad41198ac1"} Nov 03 20:58:38.413866 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.413840 5815 scope.go:122] "RemoveContainer" containerID="fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13" Nov 03 20:58:38.425628 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.425611 5815 scope.go:122] "RemoveContainer" containerID="a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6" Nov 03 20:58:38.425928 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:38.425907 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6\": container with ID starting with a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6 not found: ID does not exist" containerID="a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6" Nov 03 20:58:38.426048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.425933 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6"} err="failed to get container status \"a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6\": rpc error: code = NotFound desc = could not find container \"a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6\": container with ID starting with a9f3d107b08f5343bbeb30a10b301a49c5310143a677d2365006f4b01ec3eef6 not found: ID does not exist" Nov 03 20:58:38.426048 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.425953 5815 scope.go:122] "RemoveContainer" containerID="fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13" Nov 03 20:58:38.426263 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:38.426245 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13\": container with ID starting with fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13 not found: ID does not exist" containerID="fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13" Nov 03 20:58:38.426325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.426269 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13"} err="failed to get container status \"fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13\": rpc error: code = NotFound desc = could not find container \"fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13\": container with ID starting with fb9c7256798d5b8602021e334fc0fe501cdc301b53ea1b34ca5e3f51d191dd13 not found: ID does not exist" Nov 03 20:58:38.426325 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.426283 5815 scope.go:122] "RemoveContainer" containerID="9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b" Nov 03 20:58:38.429500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.429471 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6"] Nov 03 20:58:38.435408 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.435385 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-745c52-openshift-default-846f6fbb79-j87c6"] Nov 03 20:58:38.439479 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.439465 5815 scope.go:122] "RemoveContainer" containerID="9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b" Nov 03 20:58:38.439772 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:38.439753 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b\": container with ID starting with 9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b not found: ID does not exist" containerID="9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b" Nov 03 20:58:38.439832 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.439777 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b"} err="failed to get container status \"9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b\": rpc error: code = NotFound desc = could not find container \"9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b\": container with ID starting with 9bddcb34065fbbeba92c158efbe6a3f97a2707e673df04852554787e3b659d6b not found: ID does not exist" Nov 03 20:58:38.445253 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.445235 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6"] Nov 03 20:58:38.451721 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.451703 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-03c633-bf68b8f86-2bgg6"] Nov 03 20:58:38.456547 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.456526 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.456643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.456549 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.456643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.456564 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2483d612-5b6d-43e6-936d-8a131aff8ce9-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.456643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.456581 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmgn\" (UniqueName: \"kubernetes.io/projected/d2f8860d-3df3-4242-b9d5-bec04b4699aa-kube-api-access-dgmgn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.456643 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.456598 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ftc\" (UniqueName: \"kubernetes.io/projected/2483d612-5b6d-43e6-936d-8a131aff8ce9-kube-api-access-q8ftc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:38.591576 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.591449 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" path="/var/lib/kubelet/pods/2483d612-5b6d-43e6-936d-8a131aff8ce9/volumes" Nov 03 20:58:38.591983 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:38.591965 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f8860d-3df3-4242-b9d5-bec04b4699aa" path="/var/lib/kubelet/pods/d2f8860d-3df3-4242-b9d5-bec04b4699aa/volumes" Nov 03 20:58:40.376537 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:40.375749 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:58:41.024979 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:41.024947 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m"] Nov 03 20:58:41.028723 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:41.028699 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7aad716f-zwt7m"] Nov 03 20:58:42.667581 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:42.666705 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21519c85-ce0b-436a-b195-a454392392e9" path="/var/lib/kubelet/pods/21519c85-ce0b-436a-b195-a454392392e9/volumes" Nov 03 20:58:43.855539 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:43.855489 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:58:48.903379 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:48.903347 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 20:58:50.352856 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:50.352812 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:58:53.891500 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:53.891461 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99"] Nov 03 20:58:53.892485 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:53.892432 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" containerID="cri-o://9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545" gracePeriod=30 Nov 03 20:58:58.261380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.261355 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:58:58.359644 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.359616 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj\" (UniqueName: \"kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj\") pod \"7f35d794-158d-43b4-84b6-98fce5409ab8\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " Nov 03 20:58:58.359817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.359799 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume\") pod \"7f35d794-158d-43b4-84b6-98fce5409ab8\" (UID: \"7f35d794-158d-43b4-84b6-98fce5409ab8\") " Nov 03 20:58:58.360183 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.360156 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume" pod "7f35d794-158d-43b4-84b6-98fce5409ab8" (UID: "7f35d794-158d-43b4-84b6-98fce5409ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:58.360329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.360317 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f35d794-158d-43b4-84b6-98fce5409ab8-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:58.362176 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.362150 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj" pod "7f35d794-158d-43b4-84b6-98fce5409ab8" (UID: "7f35d794-158d-43b4-84b6-98fce5409ab8"). InnerVolumeSpecName "kube-api-access-zlbnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:58.461390 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.461366 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbnj\" (UniqueName: \"kubernetes.io/projected/7f35d794-158d-43b4-84b6-98fce5409ab8-kube-api-access-zlbnj\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:58.655614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.653248 5815 generic.go:356] "Generic (PLEG): container finished" podID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerID="58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752" exitCode=0 Nov 03 20:58:58.655614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.653646 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" event={"ID":"7f35d794-158d-43b4-84b6-98fce5409ab8","Type":"ContainerDied","Data":"58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752"} Nov 03 20:58:58.655614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.653702 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" event={"ID":"7f35d794-158d-43b4-84b6-98fce5409ab8","Type":"ContainerDied","Data":"0f648ffbaeb5e24dbd9e261da0e0b1ad22bc18fd082b5b979853712728429eca"} Nov 03 20:58:58.655614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.653731 5815 scope.go:122] "RemoveContainer" containerID="58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752" Nov 03 20:58:58.655614 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.653898 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb" Nov 03 20:58:58.672398 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.672364 5815 scope.go:122] "RemoveContainer" containerID="58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752" Nov 03 20:58:58.672782 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:58.672758 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752\": container with ID starting with 58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752 not found: ID does not exist" containerID="58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752" Nov 03 20:58:58.672926 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.672788 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752"} err="failed to get container status \"58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752\": rpc error: code = NotFound desc = could not find container \"58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752\": container with ID starting with 58be5b6967b09874bc99c53cc6a09f6be59cab5f23bde75040ff6d7d90292752 not found: ID does not exist" Nov 03 20:58:58.683988 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.683962 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb"] Nov 03 20:58:58.692651 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:58.692614 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-03c633-69568b484c-b9xqb"] Nov 03 20:58:59.183130 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.183071 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:58:59.291536 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.286577 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.300963 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301014 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301168 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301346 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301375 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301466 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301532 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.301586 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket\") pod \"633a0867-97aa-42ba-a99c-22ec46dc89b8\" (UID: \"633a0867-97aa-42ba-a99c-22ec46dc89b8\") " Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.307485 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.308888 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.311695 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.311741 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.312064 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.312324 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:59.317551 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.317164 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:59.327744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.321563 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "kube-api-access-zp6xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:59.327744 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.326760 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token" pod "633a0867-97aa-42ba-a99c-22ec46dc89b8" (UID: "633a0867-97aa-42ba-a99c-22ec46dc89b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:59.403595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403559 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403584 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403595 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403595 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403607 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403615 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403624 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633a0867-97aa-42ba-a99c-22ec46dc89b8-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403632 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403640 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633a0867-97aa-42ba-a99c-22ec46dc89b8-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.403817 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.403650 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6xd\" (UniqueName: \"kubernetes.io/projected/633a0867-97aa-42ba-a99c-22ec46dc89b8-kube-api-access-zp6xd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:59.668117 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.668077 5815 generic.go:356] "Generic (PLEG): container finished" podID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerID="9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545" exitCode=0 Nov 03 20:58:59.668288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.668149 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" Nov 03 20:58:59.668288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.668165 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" event={"ID":"633a0867-97aa-42ba-a99c-22ec46dc89b8","Type":"ContainerDied","Data":"9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545"} Nov 03 20:58:59.668288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.668225 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99" event={"ID":"633a0867-97aa-42ba-a99c-22ec46dc89b8","Type":"ContainerDied","Data":"3884a0337b673378da68f4c6075b435f4e787c9bdd818cf7c41123e9c2c8cfc1"} Nov 03 20:58:59.668288 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.668254 5815 scope.go:122] "RemoveContainer" containerID="9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545" Nov 03 20:58:59.689826 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.689808 5815 scope.go:122] "RemoveContainer" containerID="ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10" Nov 03 20:58:59.697697 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.697672 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99"] Nov 03 20:58:59.706206 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.706183 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ecee8-openshift-default-8684dc5955-mvt99"] Nov 03 20:58:59.706380 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.706367 5815 scope.go:122] "RemoveContainer" containerID="9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545" Nov 03 20:58:59.706744 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:59.706719 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545\": container with ID starting with 9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545 not found: ID does not exist" containerID="9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545" Nov 03 20:58:59.706880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.706752 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545"} err="failed to get container status \"9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545\": rpc error: code = NotFound desc = could not find container \"9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545\": container with ID starting with 9ad8f203cc18f5765a45dd81354f1ec1b9b1e4322ede7671993af42fed2f2545 not found: ID does not exist" Nov 03 20:58:59.706880 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.706774 5815 scope.go:122] "RemoveContainer" containerID="ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10" Nov 03 20:58:59.707059 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:58:59.707039 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10\": container with ID starting with ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10 not found: ID does not exist" containerID="ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10" Nov 03 20:58:59.707119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:58:59.707068 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10"} err="failed to get container status \"ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10\": rpc error: code = NotFound desc = could not find container \"ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10\": container with ID starting with ca5f2da915f80d0ab43e8aa97de8d81d961703b0e4d09338b214f253c0196f10 not found: ID does not exist" Nov 03 20:59:00.588079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:00.588038 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" path="/var/lib/kubelet/pods/633a0867-97aa-42ba-a99c-22ec46dc89b8/volumes" Nov 03 20:59:00.588584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:00.588463 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" path="/var/lib/kubelet/pods/7f35d794-158d-43b4-84b6-98fce5409ab8/volumes" Nov 03 20:59:09.881197 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.881159 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6"] Nov 03 20:59:09.882868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882839 5815 cpu_manager.go:414] "containerMap: removing container" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" Nov 03 20:59:09.882868 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882869 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882898 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882906 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882917 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882924 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882937 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882945 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882955 5815 cpu_manager.go:414] "containerMap: removing container" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882962 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882982 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d2f8860d-3df3-4242-b9d5-bec04b4699aa" containerName="payload-processing" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.882989 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f8860d-3df3-4242-b9d5-bec04b4699aa" containerName="payload-processing" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883028 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" Nov 03 20:59:09.883033 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883036 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883063 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerName="payload-pre-processing" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883070 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerName="payload-pre-processing" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883084 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerName="payload-pre-processing" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883091 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerName="payload-pre-processing" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883242 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ae31d3-cee4-44d9-a25c-381f9f1a5472" containerName="main" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883258 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883268 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883280 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e375507-e16f-4bb4-8187-39e26e4e5348" containerName="istio-proxy" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883289 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883301 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f8860d-3df3-4242-b9d5-bec04b4699aa" containerName="payload-processing" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883317 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2483d612-5b6d-43e6-936d-8a131aff8ce9" containerName="istio-proxy" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883330 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee02d163-eb9c-40e4-89a9-687da6fde6bf" containerName="payload-pre-processing" Nov 03 20:59:09.883436 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.883340 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f35d794-158d-43b4-84b6-98fce5409ab8" containerName="payload-pre-processing" Nov 03 20:59:09.891329 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.890222 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:09.894098 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.894068 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6"] Nov 03 20:59:09.899278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.897312 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-63fde4\"/\"kube-root-ca.crt\"" Nov 03 20:59:09.899278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.897417 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-63fde4\"/\"default-dockercfg-gn6zx\"" Nov 03 20:59:09.899278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.897556 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-63fde4\"/\"rate-test-model-63fde4-kserve-self-signed-certs\"" Nov 03 20:59:09.995063 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:09.995031 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-63fde4\"/\"openshift-service-ca.crt\"" Nov 03 20:59:10.015101 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.015061 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.015296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.015148 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.015296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.015194 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.015296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.015243 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.015296 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.015264 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.015568 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.015303 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gxw\" (UniqueName: \"kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.117532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.116553 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.117532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.116623 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.117532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.116664 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.117532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.116730 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.117532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.116762 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.117532 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.116816 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gxw\" (UniqueName: \"kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.118082 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.117756 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.121528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.118353 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.121528 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.120917 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.142412 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.142340 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gxw\" (UniqueName: \"kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.159561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.145107 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.159561 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.156101 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm\") pod \"rate-test-model-63fde4-kserve-6ccb477c-fcgt6\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.227533 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.226777 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:10.604441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.604415 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6"] Nov 03 20:59:10.605898 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:59:10.605864 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod278413d0_ca91_4575_8120_a1514610eaf4.slice/crio-78a40f4b4616f4a1dc9cbfcba9a760e050c48108ce27242d0137fc9ff39a2a9f WatchSource:0}: Error finding container 78a40f4b4616f4a1dc9cbfcba9a760e050c48108ce27242d0137fc9ff39a2a9f: Status 404 returned error can't find the container with id 78a40f4b4616f4a1dc9cbfcba9a760e050c48108ce27242d0137fc9ff39a2a9f Nov 03 20:59:10.799966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.799924 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" event={"ID":"278413d0-ca91-4575-8120-a1514610eaf4","Type":"ContainerStarted","Data":"d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e"} Nov 03 20:59:10.799966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.799973 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" event={"ID":"278413d0-ca91-4575-8120-a1514610eaf4","Type":"ContainerStarted","Data":"78a40f4b4616f4a1dc9cbfcba9a760e050c48108ce27242d0137fc9ff39a2a9f"} Nov 03 20:59:10.822402 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:10.822356 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" podStartSLOduration=1.822342575 podStartE2EDuration="1.822342575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:10.820048361 +0000 UTC m=+3460.350394032" watchObservedRunningTime="2026-11-03 20:59:10.822342575 +0000 UTC m=+3460.352688246" Nov 03 20:59:11.228087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:11.228057 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:13.888043 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:59:13.888005 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:59:18.711195 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:18.711163 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn"] Nov 03 20:59:20.235278 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:20.234307 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:20.264301 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:20.264268 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:20.933608 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:20.933579 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 20:59:23.711972 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:23.711934 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" podUID="1cece6e8-e19d-4b4c-b121-43e69355418d" containerName="payload-pre-processing" containerID="cri-o://556a9d5789fbbf72315e6d4f8cd483f59fe4fb21f3feeedac78389afd2f4e220" gracePeriod=25 Nov 03 20:59:23.952225 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:23.952194 5815 generic.go:356] "Generic (PLEG): container finished" podID="1cece6e8-e19d-4b4c-b121-43e69355418d" containerID="556a9d5789fbbf72315e6d4f8cd483f59fe4fb21f3feeedac78389afd2f4e220" exitCode=0 Nov 03 20:59:23.952393 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:23.952271 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" event={"ID":"1cece6e8-e19d-4b4c-b121-43e69355418d","Type":"ContainerDied","Data":"556a9d5789fbbf72315e6d4f8cd483f59fe4fb21f3feeedac78389afd2f4e220"} Nov 03 20:59:24.036468 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.036442 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:59:24.189337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.189274 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn\" (UniqueName: \"kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn\") pod \"1cece6e8-e19d-4b4c-b121-43e69355418d\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " Nov 03 20:59:24.189569 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.189420 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume\") pod \"1cece6e8-e19d-4b4c-b121-43e69355418d\" (UID: \"1cece6e8-e19d-4b4c-b121-43e69355418d\") " Nov 03 20:59:24.189904 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.189879 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume" pod "1cece6e8-e19d-4b4c-b121-43e69355418d" (UID: "1cece6e8-e19d-4b4c-b121-43e69355418d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:24.190217 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.190160 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cece6e8-e19d-4b4c-b121-43e69355418d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:24.191798 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.191769 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn" pod "1cece6e8-e19d-4b4c-b121-43e69355418d" (UID: "1cece6e8-e19d-4b4c-b121-43e69355418d"). InnerVolumeSpecName "kube-api-access-4jfnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:24.292710 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.292636 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jfnn\" (UniqueName: \"kubernetes.io/projected/1cece6e8-e19d-4b4c-b121-43e69355418d-kube-api-access-4jfnn\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:24.961718 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.961690 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" Nov 03 20:59:24.962219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.961695 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn" event={"ID":"1cece6e8-e19d-4b4c-b121-43e69355418d","Type":"ContainerDied","Data":"09b9ad46d0569fc57abead6b3dcc739edb99e8a53b7ef2a7f15e2ef01992479a"} Nov 03 20:59:24.962219 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.961831 5815 scope.go:122] "RemoveContainer" containerID="556a9d5789fbbf72315e6d4f8cd483f59fe4fb21f3feeedac78389afd2f4e220" Nov 03 20:59:24.983472 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.983433 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn"] Nov 03 20:59:24.987663 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:24.987637 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-30e9de-6d499fc695-gjhzn"] Nov 03 20:59:26.258067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.258032 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl"] Nov 03 20:59:26.259058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.259023 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1cece6e8-e19d-4b4c-b121-43e69355418d" containerName="payload-pre-processing" Nov 03 20:59:26.259058 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.259055 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cece6e8-e19d-4b4c-b121-43e69355418d" containerName="payload-pre-processing" Nov 03 20:59:26.259290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.259072 5815 cpu_manager.go:414] "containerMap: removing container" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" Nov 03 20:59:26.259290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.259078 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" Nov 03 20:59:26.259290 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.259279 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cece6e8-e19d-4b4c-b121-43e69355418d" containerName="payload-pre-processing" Nov 03 20:59:26.259464 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.259296 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="633a0867-97aa-42ba-a99c-22ec46dc89b8" containerName="istio-proxy" Nov 03 20:59:26.287572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.265743 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.287572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.276039 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3bcc2b\"/\"kube-root-ca.crt\"" Nov 03 20:59:26.287572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.276122 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-3bcc2b\"/\"openshift-service-ca.crt\"" Nov 03 20:59:26.287572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.276151 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3bcc2b\"/\"rate-test-model-3bcc2b-kserve-self-signed-certs\"" Nov 03 20:59:26.287572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.276592 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-3bcc2b\"/\"default-dockercfg-xvr46\"" Nov 03 20:59:26.287572 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.278722 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl"] Nov 03 20:59:26.441590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.435596 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.441590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.435648 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.441590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.435805 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.441590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.435862 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hnvl\" (UniqueName: \"kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.441590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.435892 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.441590 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.435918 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.537394 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.537286 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.539318 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.539289 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hnvl\" (UniqueName: \"kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.539998 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.539927 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.540605 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.540586 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.541552 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.541529 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.542087 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.542068 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.545354 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.541947 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.545679 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.541216 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.545816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.538150 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.555634 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.554925 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.558223 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.558185 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hnvl\" (UniqueName: \"kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.562017 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.561992 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs\") pod \"rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.593067 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.593035 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cece6e8-e19d-4b4c-b121-43e69355418d" path="/var/lib/kubelet/pods/1cece6e8-e19d-4b4c-b121-43e69355418d/volumes" Nov 03 20:59:26.609687 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.609643 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:26.995845 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:26.995806 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl"] Nov 03 20:59:28.002567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:28.002516 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" event={"ID":"70c47ff1-8f40-4082-bb80-b2752b747dfd","Type":"ContainerStarted","Data":"1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2"} Nov 03 20:59:28.002567 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:28.002566 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" event={"ID":"70c47ff1-8f40-4082-bb80-b2752b747dfd","Type":"ContainerStarted","Data":"a5284037aeb3647ed3ef213375e76760ac4f4568669beb29b47882c466412906"} Nov 03 20:59:28.023337 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:28.023285 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" podStartSLOduration=2.023271063 podStartE2EDuration="2.023271063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:28.022117706 +0000 UTC m=+3477.552463377" watchObservedRunningTime="2026-11-03 20:59:28.023271063 +0000 UTC m=+3477.553616736" Nov 03 20:59:28.610860 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:28.610767 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:30.026539 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:30.026467 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn"] Nov 03 20:59:30.030122 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:30.030096 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c12381-64zbn"] Nov 03 20:59:30.588754 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:30.588724 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1" path="/var/lib/kubelet/pods/ec6cdf8c-3a4c-4b0b-8af5-fd6d1196ebc1/volumes" Nov 03 20:59:36.610930 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:36.610895 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:36.624445 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:36.624415 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:37.121863 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:37.121825 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 20:59:43.853925 ip-10-0-143-81 kubenswrapper[5815]: E1103 20:59:43.853883 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 20:59:43.979159 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:43.979127 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq"] Nov 03 20:59:43.983778 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:43.983754 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:43.988119 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:43.988093 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:59:43.988352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:43.988226 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:59:43.992311 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:43.992284 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq"] Nov 03 20:59:44.063558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.053116 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9"] Nov 03 20:59:44.063558 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.058666 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-940bfa73-t62b9"] Nov 03 20:59:44.071816 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.071782 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.072088 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.072066 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6m4\" (UniqueName: \"kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.072186 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.072118 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.173600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.173498 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6m4\" (UniqueName: \"kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.173600 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.173570 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.173838 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.173684 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.174248 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.174220 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.176261 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.176242 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.185932 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.185905 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6m4\" (UniqueName: \"kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4\") pod \"maas-api-revoke-sub-db0ce8b6-bdksq\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.324072 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.324044 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:44.590151 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.590120 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0deab220-780b-4987-9c82-66f668773c95" path="/var/lib/kubelet/pods/0deab220-780b-4987-9c82-66f668773c95/volumes" Nov 03 20:59:44.615953 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:59:44.615186 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbcdc2a_cc2c_46b4_84d5_e53a136fd066.slice/crio-58ed9f4407754546dd71e02a51eef696304e7e1cc570fe4f94383e50f9d76ec6 WatchSource:0}: Error finding container 58ed9f4407754546dd71e02a51eef696304e7e1cc570fe4f94383e50f9d76ec6: Status 404 returned error can't find the container with id 58ed9f4407754546dd71e02a51eef696304e7e1cc570fe4f94383e50f9d76ec6 Nov 03 20:59:44.615953 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:44.615570 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq"] Nov 03 20:59:45.186040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:45.185941 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" event={"ID":"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066","Type":"ContainerStarted","Data":"83d7ee4fc9cd90f83dc157a7f803dd6c9faa8529cd88744512b52c518bf8b228"} Nov 03 20:59:45.186040 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:45.186009 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" event={"ID":"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066","Type":"ContainerStarted","Data":"58ed9f4407754546dd71e02a51eef696304e7e1cc570fe4f94383e50f9d76ec6"} Nov 03 20:59:45.204842 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:45.204769 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" podStartSLOduration=2.204751552 podStartE2EDuration="2.204751552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:45.204425359 +0000 UTC m=+3494.734771032" watchObservedRunningTime="2026-11-03 20:59:45.204751552 +0000 UTC m=+3494.735097225" Nov 03 20:59:46.197829 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:46.197780 5815 generic.go:356] "Generic (PLEG): container finished" podID="7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" containerID="83d7ee4fc9cd90f83dc157a7f803dd6c9faa8529cd88744512b52c518bf8b228" exitCode=0 Nov 03 20:59:46.198342 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:46.197858 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" event={"ID":"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066","Type":"ContainerDied","Data":"83d7ee4fc9cd90f83dc157a7f803dd6c9faa8529cd88744512b52c518bf8b228"} Nov 03 20:59:46.604079 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:46.604046 5815 scope.go:122] "RemoveContainer" containerID="03e27adf8af20b3a421cd18794f2a9b6afa76278df644b345dd6ca427564f4c2" Nov 03 20:59:46.616936 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:46.616914 5815 scope.go:122] "RemoveContainer" containerID="ce7ce7418ad1438e3bcea36ea7645d6ae94a872f1518516ee726ec1f2cf5307f" Nov 03 20:59:46.626966 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:46.626946 5815 scope.go:122] "RemoveContainer" containerID="0d6aef1f211ca0174fe2615920f8eec811f5089881d56907cdb54c93bc24421b" Nov 03 20:59:47.380591 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.380567 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:47.428637 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.428611 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4\" (UniqueName: \"kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4\") pod \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " Nov 03 20:59:47.428824 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.428730 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca\") pod \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " Nov 03 20:59:47.428974 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.428954 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca\") pod \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\" (UID: \"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066\") " Nov 03 20:59:47.429133 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.429112 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca" pod "7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" (UID: "7dbcdc2a-cc2c-46b4-84d5-e53a136fd066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:47.429774 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.429700 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:47.431110 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.431082 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4" pod "7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" (UID: "7dbcdc2a-cc2c-46b4-84d5-e53a136fd066"). InnerVolumeSpecName "kube-api-access-fl6m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:47.431352 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.431335 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca" pod "7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" (UID: "7dbcdc2a-cc2c-46b4-84d5-e53a136fd066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:47.531482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.531442 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl6m4\" (UniqueName: \"kubernetes.io/projected/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-kube-api-access-fl6m4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:47.531482 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:47.531477 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:48.214653 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:48.214626 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" Nov 03 20:59:48.214841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:48.214664 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq" event={"ID":"7dbcdc2a-cc2c-46b4-84d5-e53a136fd066","Type":"ContainerDied","Data":"58ed9f4407754546dd71e02a51eef696304e7e1cc570fe4f94383e50f9d76ec6"} Nov 03 20:59:48.214841 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:48.214716 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ed9f4407754546dd71e02a51eef696304e7e1cc570fe4f94383e50f9d76ec6" Nov 03 20:59:48.606258 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:48.606175 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn"] Nov 03 20:59:48.620242 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:48.620212 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns"] Nov 03 20:59:55.055589 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.055553 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj"] Nov 03 20:59:55.115752 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.115714 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb19d410-tctqj"] Nov 03 20:59:55.206811 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.206777 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r"] Nov 03 20:59:55.207427 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.207414 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" containerName="revoke-keys" Nov 03 20:59:55.207476 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.207430 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" containerName="revoke-keys" Nov 03 20:59:55.207580 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.207570 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" containerName="revoke-keys" Nov 03 20:59:55.211323 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.211305 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.216538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.215892 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:59:55.216538 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.216529 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:59:55.223156 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.223130 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r"] Nov 03 20:59:55.295316 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.295285 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.295584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.295346 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.295584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.295571 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppn4\" (UniqueName: \"kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.397606 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.397482 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.397802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.397604 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.397802 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.397720 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppn4\" (UniqueName: \"kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.398313 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.398291 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.400175 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.400143 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.410441 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.410414 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppn4\" (UniqueName: \"kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4\") pod \"maas-api-revoke-sub-5069f49e-ncj9r\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.528738 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.528706 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:55.820192 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:55.820163 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r"] Nov 03 20:59:55.820655 ip-10-0-143-81 kubenswrapper[5815]: W1103 20:59:55.820624 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeea3399_bc27_4898_9452_a4ae68c0be11.slice/crio-eee35114838a2ed4975c479ace9cec36024b6f1394943d876abe88b89c64ec28 WatchSource:0}: Error finding container eee35114838a2ed4975c479ace9cec36024b6f1394943d876abe88b89c64ec28: Status 404 returned error can't find the container with id eee35114838a2ed4975c479ace9cec36024b6f1394943d876abe88b89c64ec28 Nov 03 20:59:56.322480 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:56.322442 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" event={"ID":"deea3399-bc27-4898-9452-a4ae68c0be11","Type":"ContainerStarted","Data":"d45dfb7a7bd4d6673ef60630efb37e59adc54df5bbb2923d5f734c467c976d47"} Nov 03 20:59:56.322947 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:56.322528 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" event={"ID":"deea3399-bc27-4898-9452-a4ae68c0be11","Type":"ContainerStarted","Data":"eee35114838a2ed4975c479ace9cec36024b6f1394943d876abe88b89c64ec28"} Nov 03 20:59:56.374922 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:56.374811 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" podStartSLOduration=1.374797179 podStartE2EDuration="1.374797179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:56.370151658 +0000 UTC m=+3505.900497329" watchObservedRunningTime="2026-11-03 20:59:56.374797179 +0000 UTC m=+3505.905142848" Nov 03 20:59:56.593108 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:56.592985 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c09163f3-ab5c-44d7-a762-e195fa429018" path="/var/lib/kubelet/pods/c09163f3-ab5c-44d7-a762-e195fa429018/volumes" Nov 03 20:59:57.358584 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:57.358544 5815 generic.go:356] "Generic (PLEG): container finished" podID="deea3399-bc27-4898-9452-a4ae68c0be11" containerID="d45dfb7a7bd4d6673ef60630efb37e59adc54df5bbb2923d5f734c467c976d47" exitCode=0 Nov 03 20:59:57.359081 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:57.358613 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" event={"ID":"deea3399-bc27-4898-9452-a4ae68c0be11","Type":"ContainerDied","Data":"d45dfb7a7bd4d6673ef60630efb37e59adc54df5bbb2923d5f734c467c976d47"} Nov 03 20:59:58.541237 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.541213 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:58.658950 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.658848 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca\") pod \"deea3399-bc27-4898-9452-a4ae68c0be11\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " Nov 03 20:59:58.659199 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.659179 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4\" (UniqueName: \"kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4\") pod \"deea3399-bc27-4898-9452-a4ae68c0be11\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " Nov 03 20:59:58.659294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.659218 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca" pod "deea3399-bc27-4898-9452-a4ae68c0be11" (UID: "deea3399-bc27-4898-9452-a4ae68c0be11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:58.659294 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.659258 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca\") pod \"deea3399-bc27-4898-9452-a4ae68c0be11\" (UID: \"deea3399-bc27-4898-9452-a4ae68c0be11\") " Nov 03 20:59:58.659879 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.659856 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:58.661682 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.661657 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca" pod "deea3399-bc27-4898-9452-a4ae68c0be11" (UID: "deea3399-bc27-4898-9452-a4ae68c0be11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:58.661799 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.661767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4" pod "deea3399-bc27-4898-9452-a4ae68c0be11" (UID: "deea3399-bc27-4898-9452-a4ae68c0be11"). InnerVolumeSpecName "kube-api-access-5ppn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:58.763049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.762992 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppn4\" (UniqueName: \"kubernetes.io/projected/deea3399-bc27-4898-9452-a4ae68c0be11-kube-api-access-5ppn4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:58.763049 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:58.763029 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/deea3399-bc27-4898-9452-a4ae68c0be11-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:59.380013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:59.379979 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" Nov 03 20:59:59.380013 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:59.380001 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r" event={"ID":"deea3399-bc27-4898-9452-a4ae68c0be11","Type":"ContainerDied","Data":"eee35114838a2ed4975c479ace9cec36024b6f1394943d876abe88b89c64ec28"} Nov 03 20:59:59.380280 ip-10-0-143-81 kubenswrapper[5815]: I1103 20:59:59.380048 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eee35114838a2ed4975c479ace9cec36024b6f1394943d876abe88b89c64ec28" Nov 03 21:00:00.149060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.149025 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr"] Nov 03 21:00:00.150179 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.150155 5815 cpu_manager.go:414] "containerMap: removing container" podUID="deea3399-bc27-4898-9452-a4ae68c0be11" containerName="revoke-keys" Nov 03 21:00:00.150242 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.150183 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deea3399-bc27-4898-9452-a4ae68c0be11" containerName="revoke-keys" Nov 03 21:00:00.150338 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.150326 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="deea3399-bc27-4898-9452-a4ae68c0be11" containerName="revoke-keys" Nov 03 21:00:00.154596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.154574 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x"] Nov 03 21:00:00.154754 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.154710 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.159600 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.159581 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5"] Nov 03 21:00:00.159714 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.159695 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.162994 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.162959 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:00:00.164423 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.164386 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g"] Nov 03 21:00:00.213558 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.164693 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.226888 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.226855 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.235712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.235683 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn"] Nov 03 21:00:00.258217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.253968 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x"] Nov 03 21:00:00.258217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.254165 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.261279 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.261255 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr"] Nov 03 21:00:00.280580 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.280547 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g"] Nov 03 21:00:00.287861 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.287841 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5"] Nov 03 21:00:00.296587 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.296562 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn"] Nov 03 21:00:00.301479 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.301450 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs"] Nov 03 21:00:00.303772 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.303374 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.303772 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.303442 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.304679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.303983 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lcd\" (UniqueName: \"kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.304679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.304092 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjf4\" (UniqueName: \"kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.304679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.304278 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.304679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.304315 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgk26\" (UniqueName: \"kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.318150 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.317688 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c"] Nov 03 21:00:00.319652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.318343 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.323459 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.323440 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2"] Nov 03 21:00:00.323739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.323720 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.329285 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.329268 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw"] Nov 03 21:00:00.329440 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.329428 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.333797 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.333780 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk"] Nov 03 21:00:00.333911 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.333890 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.338163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.338139 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2"] Nov 03 21:00:00.338269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.338176 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw"] Nov 03 21:00:00.338269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.338247 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.340001 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.339981 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c"] Nov 03 21:00:00.345420 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.345403 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc"] Nov 03 21:00:00.351050 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.351030 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs"] Nov 03 21:00:00.351143 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.351099 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.354960 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.354943 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk"] Nov 03 21:00:00.358878 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.358861 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp"] Nov 03 21:00:00.364890 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.364872 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.365919 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.365900 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc"] Nov 03 21:00:00.371392 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.371371 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp"] Nov 03 21:00:00.376761 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.376744 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg"] Nov 03 21:00:00.381700 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.381682 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.382191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.382174 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg"] Nov 03 21:00:00.386601 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.386578 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f"] Nov 03 21:00:00.393557 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.391778 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.395915 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.395893 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f"] Nov 03 21:00:00.405477 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405417 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.405579 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405484 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.405639 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgk26\" (UniqueName: \"kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.405692 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405668 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xhc\" (UniqueName: \"kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.405752 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405712 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.405810 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405780 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.405864 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405837 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lcd\" (UniqueName: \"kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.405976 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.405935 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjf4\" (UniqueName: \"kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.406241 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.406217 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.406589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.406569 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.406753 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.406732 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.406834 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.406812 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.406980 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.406961 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jjh\" (UniqueName: \"kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.419309 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.419283 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgk26\" (UniqueName: \"kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.419500 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.419382 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lcd\" (UniqueName: \"kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd\") pod \"maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.420451 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.420422 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjf4\" (UniqueName: \"kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.508402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508363 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np6k5\" (UniqueName: \"kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.508633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508437 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.508633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508542 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.508633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508612 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rds6m\" (UniqueName: \"kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.508848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508659 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.508848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508702 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtgp\" (UniqueName: \"kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.508848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508733 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.508848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508829 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdv75\" (UniqueName: \"kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.509072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508866 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xqc\" (UniqueName: \"kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.509072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.508890 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.509072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509030 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.509072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509056 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.509294 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509076 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.509294 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509117 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqjz\" (UniqueName: \"kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.509294 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509253 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzdr9\" (UniqueName: \"kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.509469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509301 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq75z\" (UniqueName: \"kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.509469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509405 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jjh\" (UniqueName: \"kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.509469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509461 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.509677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509500 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcddt\" (UniqueName: \"kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.509677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509612 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.509790 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509693 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.509790 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509757 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xhc\" (UniqueName: \"kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.509790 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.509778 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.510396 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.510371 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.522112 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.522087 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jjh\" (UniqueName: \"kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh\") pod \"maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.522243 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.522113 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xhc\" (UniqueName: \"kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc\") pod \"maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.558574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.558549 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:00.570031 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.570009 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:00.598329 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.598303 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:00.610983 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.610949 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np6k5\" (UniqueName: \"kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.611217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611018 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.611217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611049 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.611217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611109 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rds6m\" (UniqueName: \"kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.611217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611150 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.611217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611203 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtgp\" (UniqueName: \"kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611248 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611284 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdv75\" (UniqueName: \"kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611312 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xqc\" (UniqueName: \"kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611337 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611408 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611435 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611459 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqjz\" (UniqueName: \"kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611489 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzdr9\" (UniqueName: \"kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.611589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611577 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq75z\" (UniqueName: \"kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.612117 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611658 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.612117 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611692 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcddt\" (UniqueName: \"kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.612117 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.611746 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.613422 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.613392 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.613599 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.612527 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.614136 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.614102 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.614280 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.614222 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.614849 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.614824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.615183 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.615160 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.615461 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.615438 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.616066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.616039 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.616630 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.616582 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.623384 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.623357 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:00.626206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.625568 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np6k5\" (UniqueName: \"kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5\") pod \"maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.626206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.626017 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:00.631771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.630080 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xqc\" (UniqueName: \"kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.631771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.630735 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdv75\" (UniqueName: \"kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.631771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.630944 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqjz\" (UniqueName: \"kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz\") pod \"maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.631771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.631545 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtgp\" (UniqueName: \"kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp\") pod \"maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.632096 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.632071 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzdr9\" (UniqueName: \"kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.635045 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.635023 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rds6m\" (UniqueName: \"kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:00.636644 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.636589 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcddt\" (UniqueName: \"kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt\") pod \"maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.640370 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.640346 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:00.644226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.644181 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq75z\" (UniqueName: \"kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.654805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.654347 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:00.659716 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.659694 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:00.681618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.680963 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:00.685206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.685148 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:00.689665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.689460 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:00.710611 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.697952 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:00.710611 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.708877 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:00.722912 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:00.715819 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:01.142604 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.142522 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x"] Nov 03 21:00:01.157051 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.155803 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr"] Nov 03 21:00:01.180920 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.178625 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5"] Nov 03 21:00:01.368745 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.368718 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg"] Nov 03 21:00:01.372158 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.372126 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191ab3e9_ce49_4655_830a_7dc8160e9749.slice/crio-58b8c4122411e944c77944fe07deb80f22d0c06783a4cd060b36197ab986ea3e WatchSource:0}: Error finding container 58b8c4122411e944c77944fe07deb80f22d0c06783a4cd060b36197ab986ea3e: Status 404 returned error can't find the container with id 58b8c4122411e944c77944fe07deb80f22d0c06783a4cd060b36197ab986ea3e Nov 03 21:00:01.373759 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.373740 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g"] Nov 03 21:00:01.374547 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.374489 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1863d0ed_1093_4ba8_adbe_bb0ce554ff8e.slice/crio-2d69e30d4c3ca2816393f8a48d9414df34548e5be138f2200e8aa5d3e64857b3 WatchSource:0}: Error finding container 2d69e30d4c3ca2816393f8a48d9414df34548e5be138f2200e8aa5d3e64857b3: Status 404 returned error can't find the container with id 2d69e30d4c3ca2816393f8a48d9414df34548e5be138f2200e8aa5d3e64857b3 Nov 03 21:00:01.405068 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.405034 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" event={"ID":"f1bcb275-c36a-4919-9526-afe5f86e8ad3","Type":"ContainerStarted","Data":"0ce8b9813738920edf63c555545168d479f234c0f0ec45342320e5c642980a91"} Nov 03 21:00:01.409773 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.409739 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" event={"ID":"191ab3e9-ce49-4655-830a-7dc8160e9749","Type":"ContainerStarted","Data":"58b8c4122411e944c77944fe07deb80f22d0c06783a4cd060b36197ab986ea3e"} Nov 03 21:00:01.411668 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.411637 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerStarted","Data":"e17e3e2ce5a409e8b7703e865744b2480095ccda98efe606b3f9b6cd14e51910"} Nov 03 21:00:01.413790 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.413730 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" event={"ID":"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e","Type":"ContainerStarted","Data":"2d69e30d4c3ca2816393f8a48d9414df34548e5be138f2200e8aa5d3e64857b3"} Nov 03 21:00:01.415226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.415203 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" event={"ID":"53905733-a798-40db-8e03-7bfc5e0152a9","Type":"ContainerStarted","Data":"d5d3b5735118afc45b1bf84ebbc9b1106607ebe54c8660cca90efd9f04ff154e"} Nov 03 21:00:01.441899 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.441866 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14c22cc_8192_4cc4_86a5_bd11e23de11d.slice/crio-370e7cf10e3846485ccd1f768dc33ccd65a3e880f2d8c93bdd5c4acbf6005684 WatchSource:0}: Error finding container 370e7cf10e3846485ccd1f768dc33ccd65a3e880f2d8c93bdd5c4acbf6005684: Status 404 returned error can't find the container with id 370e7cf10e3846485ccd1f768dc33ccd65a3e880f2d8c93bdd5c4acbf6005684 Nov 03 21:00:01.448471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.448426 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn"] Nov 03 21:00:01.475786 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.475758 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc"] Nov 03 21:00:01.476318 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.476289 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2160c8_a5c9_4d9e_a33c_d8add9c2ddc6.slice/crio-ccbf0de2702d094fe7f03dcfd30d9fd91b4ee9a9d28c495417029f21fd819fdb WatchSource:0}: Error finding container ccbf0de2702d094fe7f03dcfd30d9fd91b4ee9a9d28c495417029f21fd819fdb: Status 404 returned error can't find the container with id ccbf0de2702d094fe7f03dcfd30d9fd91b4ee9a9d28c495417029f21fd819fdb Nov 03 21:00:01.482775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.482745 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs"] Nov 03 21:00:01.525812 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.525786 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82750f5b_62f3_4282_9ca1_59613ffb45d8.slice/crio-f939e7c1999f1e4839929bfda6cdef3445d44a7e074a48b94b488b123aeda796 WatchSource:0}: Error finding container f939e7c1999f1e4839929bfda6cdef3445d44a7e074a48b94b488b123aeda796: Status 404 returned error can't find the container with id f939e7c1999f1e4839929bfda6cdef3445d44a7e074a48b94b488b123aeda796 Nov 03 21:00:01.526869 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.526844 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk"] Nov 03 21:00:01.539750 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.539724 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c"] Nov 03 21:00:01.540768 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.540744 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93056b14_35ff_4299_ae03_ce994d5f31f3.slice/crio-de33e130874eb778fbf8f111d28af737da947612d0aa63b96bfea56a3694e306 WatchSource:0}: Error finding container de33e130874eb778fbf8f111d28af737da947612d0aa63b96bfea56a3694e306: Status 404 returned error can't find the container with id de33e130874eb778fbf8f111d28af737da947612d0aa63b96bfea56a3694e306 Nov 03 21:00:01.603988 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.603767 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw"] Nov 03 21:00:01.608200 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:01.608171 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8fd06c_af59_4b1f_b8b9_8f5e26a8422e.slice/crio-2e6ffb9193fde30d6e2c36e90212d59216cc4986f6117fd5a3e2bc7562eaab71 WatchSource:0}: Error finding container 2e6ffb9193fde30d6e2c36e90212d59216cc4986f6117fd5a3e2bc7562eaab71: Status 404 returned error can't find the container with id 2e6ffb9193fde30d6e2c36e90212d59216cc4986f6117fd5a3e2bc7562eaab71 Nov 03 21:00:01.637843 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.636553 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2"] Nov 03 21:00:01.643550 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.643479 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp"] Nov 03 21:00:01.647537 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:01.647477 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f"] Nov 03 21:00:02.424392 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.424350 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" event={"ID":"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5","Type":"ContainerStarted","Data":"8b52d02b1b81beaa99b37a8d2d7051c98f08011ed415fa08eec7c7fa345f6662"} Nov 03 21:00:02.426094 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.426065 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" event={"ID":"012a9e62-3609-49b1-8741-0f30c617c8ea","Type":"ContainerStarted","Data":"0cd081e68d1829313518e5c0fafb7be5b9b7bfc03cdc25519d185df38b73b24a"} Nov 03 21:00:02.426239 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.426122 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" event={"ID":"012a9e62-3609-49b1-8741-0f30c617c8ea","Type":"ContainerStarted","Data":"0c9a5827fa42dd436d5d2f0d70bf5d2b821c289a3becc4f8d556e88ffea2b0df"} Nov 03 21:00:02.427815 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.427788 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" event={"ID":"53905733-a798-40db-8e03-7bfc5e0152a9","Type":"ContainerStarted","Data":"bbb3878dc6ae8be7ad3526767a7eb99b0b97aaacd33ae0044f730b5bfb9d68c1"} Nov 03 21:00:02.429598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.429571 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" event={"ID":"82750f5b-62f3-4282-9ca1-59613ffb45d8","Type":"ContainerStarted","Data":"5065a2220f223807aa45a460cea2283ce7003ce6dca2b46153774286d908801d"} Nov 03 21:00:02.429703 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.429619 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" event={"ID":"82750f5b-62f3-4282-9ca1-59613ffb45d8","Type":"ContainerStarted","Data":"f939e7c1999f1e4839929bfda6cdef3445d44a7e074a48b94b488b123aeda796"} Nov 03 21:00:02.431453 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.431428 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" event={"ID":"f1bcb275-c36a-4919-9526-afe5f86e8ad3","Type":"ContainerStarted","Data":"ce1ee7b7282c57fe53f203e87044bbf1027df2b8b888a0d6ea58f94c4de0dbe1"} Nov 03 21:00:02.433017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.432994 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" event={"ID":"191ab3e9-ce49-4655-830a-7dc8160e9749","Type":"ContainerStarted","Data":"eb46131212b6d1ece5f2836face0fe9cc27f9b9c00fe1d5e67f156664b4a0686"} Nov 03 21:00:02.434738 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.434714 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerStarted","Data":"f86dddebaea884033331b715ffd77f6a5ba6035a7d7a0996a07bcb43c2c67076"} Nov 03 21:00:02.434850 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.434757 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerStarted","Data":"ccbf0de2702d094fe7f03dcfd30d9fd91b4ee9a9d28c495417029f21fd819fdb"} Nov 03 21:00:02.436367 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.436344 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerStarted","Data":"1d468ef5dddf876f6890dfd3f44abcf411824eb64c88520eb30ba74a542117ff"} Nov 03 21:00:02.436476 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.436383 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerStarted","Data":"5b4b6308afc9420cb2a5efc05fe64c3d6cbfcc510b7378ee90ef6fd3b3702cef"} Nov 03 21:00:02.438097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.438074 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerStarted","Data":"e7692f5f5d2916002a5a4d3a20ae8015e2374e5c9a7d865439e73d962f8e8538"} Nov 03 21:00:02.439683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.439658 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" event={"ID":"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e","Type":"ContainerStarted","Data":"a90ac36f4eb3deae2e48c1d98abeac3cd80b7a24c6ecb40d6128cad2ff28a80e"} Nov 03 21:00:02.439797 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.439705 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" event={"ID":"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e","Type":"ContainerStarted","Data":"2e6ffb9193fde30d6e2c36e90212d59216cc4986f6117fd5a3e2bc7562eaab71"} Nov 03 21:00:02.441226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.441201 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" event={"ID":"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e","Type":"ContainerStarted","Data":"f26f6c3382ee3069f17ac44626266bd2435776f7ea3fb977dd365a17cbbd85ea"} Nov 03 21:00:02.442746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.442723 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerStarted","Data":"fa45f975360134dc0d7ea9289bbb5f801cd406a227d8f233df7f55a9a9299e47"} Nov 03 21:00:02.442848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.442765 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerStarted","Data":"370e7cf10e3846485ccd1f768dc33ccd65a3e880f2d8c93bdd5c4acbf6005684"} Nov 03 21:00:02.444922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.444896 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerStarted","Data":"df02b73335038ca3e405fdcc4c7becdfd90d44a1be7e1b4f1f0afd3086d1b047"} Nov 03 21:00:02.445021 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.444944 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerStarted","Data":"de33e130874eb778fbf8f111d28af737da947612d0aa63b96bfea56a3694e306"} Nov 03 21:00:02.445168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.445123 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" podStartSLOduration=2.445108524 podStartE2EDuration="2.445108524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.44362107 +0000 UTC m=+3511.973966741" watchObservedRunningTime="2026-11-03 21:00:02.445108524 +0000 UTC m=+3511.975454194" Nov 03 21:00:02.446636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.446615 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerStarted","Data":"cee4b17de921cd9f47a1e68f4be82d784dd02f90cd4a75ccbbc7dd20b37a3589"} Nov 03 21:00:02.446752 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.446655 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerStarted","Data":"6b3569578ad302eb65c27634957f6b3b8527c781763f7d5fb091a0fdaf1cac39"} Nov 03 21:00:02.477069 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.465487 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" podStartSLOduration=2.46547115 podStartE2EDuration="2.46547115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.46126153 +0000 UTC m=+3511.991607203" watchObservedRunningTime="2026-11-03 21:00:02.46547115 +0000 UTC m=+3511.995816822" Nov 03 21:00:02.491079 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.491005 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" podStartSLOduration=2.490987613 podStartE2EDuration="2.490987613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.488177022 +0000 UTC m=+3512.018522697" watchObservedRunningTime="2026-11-03 21:00:02.490987613 +0000 UTC m=+3512.021333286" Nov 03 21:00:02.507660 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.507620 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" podStartSLOduration=2.507607062 podStartE2EDuration="2.507607062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.505896593 +0000 UTC m=+3512.036242266" watchObservedRunningTime="2026-11-03 21:00:02.507607062 +0000 UTC m=+3512.037952779" Nov 03 21:00:02.527447 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.527406 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" podStartSLOduration=2.5273945380000002 podStartE2EDuration="2.527394538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.525733874 +0000 UTC m=+3512.056079544" watchObservedRunningTime="2026-11-03 21:00:02.527394538 +0000 UTC m=+3512.057740210" Nov 03 21:00:02.546258 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.546176 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" podStartSLOduration=2.546164253 podStartE2EDuration="2.546164253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.543744131 +0000 UTC m=+3512.074089805" watchObservedRunningTime="2026-11-03 21:00:02.546164253 +0000 UTC m=+3512.076509925" Nov 03 21:00:02.562895 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.562854 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" podStartSLOduration=2.562840782 podStartE2EDuration="2.562840782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.561795894 +0000 UTC m=+3512.092141566" watchObservedRunningTime="2026-11-03 21:00:02.562840782 +0000 UTC m=+3512.093186484" Nov 03 21:00:02.596483 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.596426 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" podStartSLOduration=2.596408607 podStartE2EDuration="2.596408607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.591575091 +0000 UTC m=+3512.121920763" watchObservedRunningTime="2026-11-03 21:00:02.596408607 +0000 UTC m=+3512.126754284" Nov 03 21:00:02.615850 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.615793 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" podStartSLOduration=2.6157827559999998 podStartE2EDuration="2.615782756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.614146695 +0000 UTC m=+3512.144492381" watchObservedRunningTime="2026-11-03 21:00:02.615782756 +0000 UTC m=+3512.146128437" Nov 03 21:00:02.634903 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.634862 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" podStartSLOduration=2.634849471 podStartE2EDuration="2.634849471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.631907619 +0000 UTC m=+3512.162253288" watchObservedRunningTime="2026-11-03 21:00:02.634849471 +0000 UTC m=+3512.165195137" Nov 03 21:00:02.650474 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.650438 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" podStartSLOduration=2.650426985 podStartE2EDuration="2.650426985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.649134977 +0000 UTC m=+3512.179480651" watchObservedRunningTime="2026-11-03 21:00:02.650426985 +0000 UTC m=+3512.180772657" Nov 03 21:00:02.683860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.683786 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" podStartSLOduration=2.683767259 podStartE2EDuration="2.683767259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.670036724 +0000 UTC m=+3512.200382396" watchObservedRunningTime="2026-11-03 21:00:02.683767259 +0000 UTC m=+3512.214112933" Nov 03 21:00:02.697198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:02.697152 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" podStartSLOduration=2.697139606 podStartE2EDuration="2.697139606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.696319072 +0000 UTC m=+3512.226664744" watchObservedRunningTime="2026-11-03 21:00:02.697139606 +0000 UTC m=+3512.227485278" Nov 03 21:00:03.486200 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.486158 5815 generic.go:356] "Generic (PLEG): container finished" podID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerID="1d468ef5dddf876f6890dfd3f44abcf411824eb64c88520eb30ba74a542117ff" exitCode=60 Nov 03 21:00:03.486763 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.486241 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerDied","Data":"1d468ef5dddf876f6890dfd3f44abcf411824eb64c88520eb30ba74a542117ff"} Nov 03 21:00:03.486763 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.486595 5815 scope.go:122] "RemoveContainer" containerID="1d468ef5dddf876f6890dfd3f44abcf411824eb64c88520eb30ba74a542117ff" Nov 03 21:00:03.488912 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.488558 5815 generic.go:356] "Generic (PLEG): container finished" podID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerID="e7692f5f5d2916002a5a4d3a20ae8015e2374e5c9a7d865439e73d962f8e8538" exitCode=60 Nov 03 21:00:03.488912 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.488674 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerDied","Data":"e7692f5f5d2916002a5a4d3a20ae8015e2374e5c9a7d865439e73d962f8e8538"} Nov 03 21:00:03.489220 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.488987 5815 scope.go:122] "RemoveContainer" containerID="e7692f5f5d2916002a5a4d3a20ae8015e2374e5c9a7d865439e73d962f8e8538" Nov 03 21:00:03.491642 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.491616 5815 generic.go:356] "Generic (PLEG): container finished" podID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerID="a90ac36f4eb3deae2e48c1d98abeac3cd80b7a24c6ecb40d6128cad2ff28a80e" exitCode=60 Nov 03 21:00:03.491788 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.491726 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" event={"ID":"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e","Type":"ContainerDied","Data":"a90ac36f4eb3deae2e48c1d98abeac3cd80b7a24c6ecb40d6128cad2ff28a80e"} Nov 03 21:00:03.491966 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.491943 5815 scope.go:122] "RemoveContainer" containerID="a90ac36f4eb3deae2e48c1d98abeac3cd80b7a24c6ecb40d6128cad2ff28a80e" Nov 03 21:00:03.494040 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.494018 5815 generic.go:356] "Generic (PLEG): container finished" podID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerID="f26f6c3382ee3069f17ac44626266bd2435776f7ea3fb977dd365a17cbbd85ea" exitCode=60 Nov 03 21:00:03.494158 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.494099 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" event={"ID":"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e","Type":"ContainerDied","Data":"f26f6c3382ee3069f17ac44626266bd2435776f7ea3fb977dd365a17cbbd85ea"} Nov 03 21:00:03.494353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.494329 5815 scope.go:122] "RemoveContainer" containerID="f26f6c3382ee3069f17ac44626266bd2435776f7ea3fb977dd365a17cbbd85ea" Nov 03 21:00:03.496141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.496113 5815 generic.go:356] "Generic (PLEG): container finished" podID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerID="fa45f975360134dc0d7ea9289bbb5f801cd406a227d8f233df7f55a9a9299e47" exitCode=60 Nov 03 21:00:03.496260 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.496201 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerDied","Data":"fa45f975360134dc0d7ea9289bbb5f801cd406a227d8f233df7f55a9a9299e47"} Nov 03 21:00:03.496414 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.496398 5815 scope.go:122] "RemoveContainer" containerID="fa45f975360134dc0d7ea9289bbb5f801cd406a227d8f233df7f55a9a9299e47" Nov 03 21:00:03.503363 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.503342 5815 generic.go:356] "Generic (PLEG): container finished" podID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerID="df02b73335038ca3e405fdcc4c7becdfd90d44a1be7e1b4f1f0afd3086d1b047" exitCode=60 Nov 03 21:00:03.503681 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.503630 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerDied","Data":"df02b73335038ca3e405fdcc4c7becdfd90d44a1be7e1b4f1f0afd3086d1b047"} Nov 03 21:00:03.506646 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.504187 5815 scope.go:122] "RemoveContainer" containerID="df02b73335038ca3e405fdcc4c7becdfd90d44a1be7e1b4f1f0afd3086d1b047" Nov 03 21:00:03.512321 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.512299 5815 generic.go:356] "Generic (PLEG): container finished" podID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerID="cee4b17de921cd9f47a1e68f4be82d784dd02f90cd4a75ccbbc7dd20b37a3589" exitCode=60 Nov 03 21:00:03.512425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.512376 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerDied","Data":"cee4b17de921cd9f47a1e68f4be82d784dd02f90cd4a75ccbbc7dd20b37a3589"} Nov 03 21:00:03.512686 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.512664 5815 scope.go:122] "RemoveContainer" containerID="cee4b17de921cd9f47a1e68f4be82d784dd02f90cd4a75ccbbc7dd20b37a3589" Nov 03 21:00:03.528373 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.528344 5815 generic.go:356] "Generic (PLEG): container finished" podID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerID="8b474e782fe864cc161ac25198c447bc627a0db5d66dc0da55961a4e4de5f5fb" exitCode=60 Nov 03 21:00:03.529219 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.529198 5815 scope.go:122] "RemoveContainer" containerID="8b474e782fe864cc161ac25198c447bc627a0db5d66dc0da55961a4e4de5f5fb" Nov 03 21:00:03.529645 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.529426 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" event={"ID":"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5","Type":"ContainerDied","Data":"8b474e782fe864cc161ac25198c447bc627a0db5d66dc0da55961a4e4de5f5fb"} Nov 03 21:00:03.536205 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.535323 5815 generic.go:356] "Generic (PLEG): container finished" podID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerID="0cd081e68d1829313518e5c0fafb7be5b9b7bfc03cdc25519d185df38b73b24a" exitCode=60 Nov 03 21:00:03.536205 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.535934 5815 scope.go:122] "RemoveContainer" containerID="0cd081e68d1829313518e5c0fafb7be5b9b7bfc03cdc25519d185df38b73b24a" Nov 03 21:00:03.536205 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.535979 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" event={"ID":"012a9e62-3609-49b1-8741-0f30c617c8ea","Type":"ContainerDied","Data":"0cd081e68d1829313518e5c0fafb7be5b9b7bfc03cdc25519d185df38b73b24a"} Nov 03 21:00:03.539352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.539328 5815 generic.go:356] "Generic (PLEG): container finished" podID="53905733-a798-40db-8e03-7bfc5e0152a9" containerID="bbb3878dc6ae8be7ad3526767a7eb99b0b97aaacd33ae0044f730b5bfb9d68c1" exitCode=60 Nov 03 21:00:03.539470 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.539453 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" event={"ID":"53905733-a798-40db-8e03-7bfc5e0152a9","Type":"ContainerDied","Data":"bbb3878dc6ae8be7ad3526767a7eb99b0b97aaacd33ae0044f730b5bfb9d68c1"} Nov 03 21:00:03.540029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.540004 5815 scope.go:122] "RemoveContainer" containerID="bbb3878dc6ae8be7ad3526767a7eb99b0b97aaacd33ae0044f730b5bfb9d68c1" Nov 03 21:00:03.549291 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.549265 5815 generic.go:356] "Generic (PLEG): container finished" podID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerID="5065a2220f223807aa45a460cea2283ce7003ce6dca2b46153774286d908801d" exitCode=60 Nov 03 21:00:03.549594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.549572 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" event={"ID":"82750f5b-62f3-4282-9ca1-59613ffb45d8","Type":"ContainerDied","Data":"5065a2220f223807aa45a460cea2283ce7003ce6dca2b46153774286d908801d"} Nov 03 21:00:03.550049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.550030 5815 scope.go:122] "RemoveContainer" containerID="5065a2220f223807aa45a460cea2283ce7003ce6dca2b46153774286d908801d" Nov 03 21:00:03.557064 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.556983 5815 generic.go:356] "Generic (PLEG): container finished" podID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerID="ce1ee7b7282c57fe53f203e87044bbf1027df2b8b888a0d6ea58f94c4de0dbe1" exitCode=60 Nov 03 21:00:03.557064 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.557056 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" event={"ID":"f1bcb275-c36a-4919-9526-afe5f86e8ad3","Type":"ContainerDied","Data":"ce1ee7b7282c57fe53f203e87044bbf1027df2b8b888a0d6ea58f94c4de0dbe1"} Nov 03 21:00:03.557268 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.557253 5815 scope.go:122] "RemoveContainer" containerID="ce1ee7b7282c57fe53f203e87044bbf1027df2b8b888a0d6ea58f94c4de0dbe1" Nov 03 21:00:03.559615 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.559594 5815 generic.go:356] "Generic (PLEG): container finished" podID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerID="eb46131212b6d1ece5f2836face0fe9cc27f9b9c00fe1d5e67f156664b4a0686" exitCode=60 Nov 03 21:00:03.559716 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.559646 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" event={"ID":"191ab3e9-ce49-4655-830a-7dc8160e9749","Type":"ContainerDied","Data":"eb46131212b6d1ece5f2836face0fe9cc27f9b9c00fe1d5e67f156664b4a0686"} Nov 03 21:00:03.560115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.560099 5815 scope.go:122] "RemoveContainer" containerID="eb46131212b6d1ece5f2836face0fe9cc27f9b9c00fe1d5e67f156664b4a0686" Nov 03 21:00:03.562048 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.562025 5815 generic.go:356] "Generic (PLEG): container finished" podID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerID="f86dddebaea884033331b715ffd77f6a5ba6035a7d7a0996a07bcb43c2c67076" exitCode=60 Nov 03 21:00:03.562122 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.562056 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerDied","Data":"f86dddebaea884033331b715ffd77f6a5ba6035a7d7a0996a07bcb43c2c67076"} Nov 03 21:00:03.562355 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.562337 5815 scope.go:122] "RemoveContainer" containerID="f86dddebaea884033331b715ffd77f6a5ba6035a7d7a0996a07bcb43c2c67076" Nov 03 21:00:03.630637 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.630600 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" containerID="cri-o://3caa77faedd47ce304cf81db691e43247416c42a6dc3d25cb6daf60ede9d251c" gracePeriod=45 Nov 03 21:00:03.648013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:03.647975 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" containerID="cri-o://6ba19fc6197a89656f8707a3c37de0b06ac4973d68593fedaae2d7434e2e19b5" gracePeriod=45 Nov 03 21:00:04.605643 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.604989 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerStarted","Data":"84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c"} Nov 03 21:00:04.613766 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.612488 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerStarted","Data":"dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b"} Nov 03 21:00:04.617439 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.616401 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerStarted","Data":"611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01"} Nov 03 21:00:04.622007 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.621184 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerStarted","Data":"922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f"} Nov 03 21:00:04.626896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.625917 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerStarted","Data":"776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334"} Nov 03 21:00:04.629818 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.629791 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerStarted","Data":"de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58"} Nov 03 21:00:04.632625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.632483 5815 generic.go:356] "Generic (PLEG): container finished" podID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerID="6ba19fc6197a89656f8707a3c37de0b06ac4973d68593fedaae2d7434e2e19b5" exitCode=0 Nov 03 21:00:04.632625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.632534 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" event={"ID":"baf361fa-7064-41ed-af8d-c674ae0eb7d1","Type":"ContainerDied","Data":"6ba19fc6197a89656f8707a3c37de0b06ac4973d68593fedaae2d7434e2e19b5"} Nov 03 21:00:04.635554 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.635532 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" event={"ID":"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5","Type":"ContainerStarted","Data":"569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b"} Nov 03 21:00:04.637766 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.637746 5815 generic.go:356] "Generic (PLEG): container finished" podID="d7781320-da95-4c12-9645-b22679a15067" containerID="3caa77faedd47ce304cf81db691e43247416c42a6dc3d25cb6daf60ede9d251c" exitCode=0 Nov 03 21:00:04.637865 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.637783 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" event={"ID":"d7781320-da95-4c12-9645-b22679a15067","Type":"ContainerDied","Data":"3caa77faedd47ce304cf81db691e43247416c42a6dc3d25cb6daf60ede9d251c"} Nov 03 21:00:04.740964 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.740923 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" podStartSLOduration=4.740909879 podStartE2EDuration="4.740909879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:04.740601432 +0000 UTC m=+3514.270947100" watchObservedRunningTime="2026-11-03 21:00:04.740909879 +0000 UTC m=+3514.271255551" Nov 03 21:00:04.742065 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.742041 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 21:00:04.746327 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.746304 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 21:00:04.806478 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806447 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home\") pod \"d7781320-da95-4c12-9645-b22679a15067\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " Nov 03 21:00:04.806604 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806549 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp\" (UniqueName: \"kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp\") pod \"d7781320-da95-4c12-9645-b22679a15067\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " Nov 03 21:00:04.806645 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806635 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm\") pod \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " Nov 03 21:00:04.806695 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806679 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache\") pod \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " Nov 03 21:00:04.806742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806726 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache\") pod \"d7781320-da95-4c12-9645-b22679a15067\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " Nov 03 21:00:04.806806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806793 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm\") pod \"d7781320-da95-4c12-9645-b22679a15067\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " Nov 03 21:00:04.806845 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806833 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs\") pod \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " Nov 03 21:00:04.806913 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806900 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home\") pod \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " Nov 03 21:00:04.806948 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.806934 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs\") pod \"d7781320-da95-4c12-9645-b22679a15067\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " Nov 03 21:00:04.807085 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.807072 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir\") pod \"d7781320-da95-4c12-9645-b22679a15067\" (UID: \"d7781320-da95-4c12-9645-b22679a15067\") " Nov 03 21:00:04.807164 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.807113 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir\") pod \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " Nov 03 21:00:04.807217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.807206 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq\" (UniqueName: \"kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq\") pod \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\" (UID: \"baf361fa-7064-41ed-af8d-c674ae0eb7d1\") " Nov 03 21:00:04.811149 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.811114 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache" pod "d7781320-da95-4c12-9645-b22679a15067" (UID: "d7781320-da95-4c12-9645-b22679a15067"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.811462 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.811439 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home" pod "d7781320-da95-4c12-9645-b22679a15067" (UID: "d7781320-da95-4c12-9645-b22679a15067"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.814481 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.814457 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq" pod "baf361fa-7064-41ed-af8d-c674ae0eb7d1" (UID: "baf361fa-7064-41ed-af8d-c674ae0eb7d1"). InnerVolumeSpecName "kube-api-access-m59dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:04.824361 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.824085 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp" pod "d7781320-da95-4c12-9645-b22679a15067" (UID: "d7781320-da95-4c12-9645-b22679a15067"). InnerVolumeSpecName "kube-api-access-z2bwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:04.825757 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.825729 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home" pod "baf361fa-7064-41ed-af8d-c674ae0eb7d1" (UID: "baf361fa-7064-41ed-af8d-c674ae0eb7d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.826115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.825958 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir" pod "d7781320-da95-4c12-9645-b22679a15067" (UID: "d7781320-da95-4c12-9645-b22679a15067"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.826425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.826272 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache" pod "baf361fa-7064-41ed-af8d-c674ae0eb7d1" (UID: "baf361fa-7064-41ed-af8d-c674ae0eb7d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.827333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.826598 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir" pod "baf361fa-7064-41ed-af8d-c674ae0eb7d1" (UID: "baf361fa-7064-41ed-af8d-c674ae0eb7d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.827333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.826917 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs" pod "d7781320-da95-4c12-9645-b22679a15067" (UID: "d7781320-da95-4c12-9645-b22679a15067"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:04.830407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.830116 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm" pod "d7781320-da95-4c12-9645-b22679a15067" (UID: "d7781320-da95-4c12-9645-b22679a15067"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.835556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.831307 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm" pod "baf361fa-7064-41ed-af8d-c674ae0eb7d1" (UID: "baf361fa-7064-41ed-af8d-c674ae0eb7d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.836965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.836938 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs" pod "baf361fa-7064-41ed-af8d-c674ae0eb7d1" (UID: "baf361fa-7064-41ed-af8d-c674ae0eb7d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:04.909569 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.909540 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bwp\" (UniqueName: \"kubernetes.io/projected/d7781320-da95-4c12-9645-b22679a15067-kube-api-access-z2bwp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910456 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910434 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910494 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910626 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910641 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910655 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910666 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910678 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7781320-da95-4c12-9645-b22679a15067-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910716 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910729 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf361fa-7064-41ed-af8d-c674ae0eb7d1-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910742 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m59dq\" (UniqueName: \"kubernetes.io/projected/baf361fa-7064-41ed-af8d-c674ae0eb7d1-kube-api-access-m59dq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.910775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:04.910757 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7781320-da95-4c12-9645-b22679a15067-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:05.281746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.281707 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 21:00:05.601424 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.601336 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:05.649564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.649534 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" event={"ID":"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e","Type":"ContainerDied","Data":"de58eac52c99175d2fd5b13de90c89327f803e1623cfcaaa65189df99ec2743b"} Nov 03 21:00:05.649564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.649435 5815 generic.go:356] "Generic (PLEG): container finished" podID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerID="de58eac52c99175d2fd5b13de90c89327f803e1623cfcaaa65189df99ec2743b" exitCode=60 Nov 03 21:00:05.650030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.649601 5815 scope.go:122] "RemoveContainer" containerID="f26f6c3382ee3069f17ac44626266bd2435776f7ea3fb977dd365a17cbbd85ea" Nov 03 21:00:05.650030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.649849 5815 scope.go:122] "RemoveContainer" containerID="de58eac52c99175d2fd5b13de90c89327f803e1623cfcaaa65189df99ec2743b" Nov 03 21:00:05.650124 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.650105 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g_odh-ai-gateway-infra(1863d0ed-1093-4ba8-adbe-bb0ce554ff8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" Nov 03 21:00:05.651760 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.651737 5815 generic.go:356] "Generic (PLEG): container finished" podID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerID="922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f" exitCode=60 Nov 03 21:00:05.651875 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.651821 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerDied","Data":"922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f"} Nov 03 21:00:05.652028 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.652014 5815 scope.go:122] "RemoveContainer" containerID="922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f" Nov 03 21:00:05.652259 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.652234 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn_odh-ai-gateway-infra(c14c22cc-8192-4cc4-86a5-bd11e23de11d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" Nov 03 21:00:05.653846 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.653822 5815 generic.go:356] "Generic (PLEG): container finished" podID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerID="776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334" exitCode=60 Nov 03 21:00:05.653957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.653896 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerDied","Data":"776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334"} Nov 03 21:00:05.654175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.654157 5815 scope.go:122] "RemoveContainer" containerID="776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334" Nov 03 21:00:05.654381 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.654363 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c_odh-ai-gateway-infra(93056b14-35ff-4299-ae03-ce994d5f31f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" Nov 03 21:00:05.656035 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.656014 5815 generic.go:356] "Generic (PLEG): container finished" podID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerID="de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58" exitCode=60 Nov 03 21:00:05.656203 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.656041 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerDied","Data":"de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58"} Nov 03 21:00:05.656336 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.656302 5815 scope.go:122] "RemoveContainer" containerID="de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58" Nov 03 21:00:05.656625 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.656598 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs_odh-ai-gateway-infra(a16e4409-5714-49f4-9af0-c36e0760fcd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" Nov 03 21:00:05.657564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.657546 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" Nov 03 21:00:05.657564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.657554 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns" event={"ID":"baf361fa-7064-41ed-af8d-c674ae0eb7d1","Type":"ContainerDied","Data":"d57e11e2fa541b20c491c3f1f7c749c98a54764421e22001198610f215a742d2"} Nov 03 21:00:05.659605 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.659586 5815 generic.go:356] "Generic (PLEG): container finished" podID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerID="569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b" exitCode=60 Nov 03 21:00:05.659730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.659642 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" event={"ID":"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5","Type":"ContainerDied","Data":"569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b"} Nov 03 21:00:05.659880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.659865 5815 scope.go:122] "RemoveContainer" containerID="569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b" Nov 03 21:00:05.660121 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.660098 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f_odh-ai-gateway-infra(c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" Nov 03 21:00:05.661748 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.661718 5815 generic.go:356] "Generic (PLEG): container finished" podID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerID="0540b8bb2657f14ae1228018ed97e235ae5b8c6b8ba3cd4ded5807c0a34b8d16" exitCode=60 Nov 03 21:00:05.661852 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.661823 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" event={"ID":"012a9e62-3609-49b1-8741-0f30c617c8ea","Type":"ContainerDied","Data":"0540b8bb2657f14ae1228018ed97e235ae5b8c6b8ba3cd4ded5807c0a34b8d16"} Nov 03 21:00:05.661996 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.661981 5815 scope.go:122] "RemoveContainer" containerID="0540b8bb2657f14ae1228018ed97e235ae5b8c6b8ba3cd4ded5807c0a34b8d16" Nov 03 21:00:05.662251 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.662228 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2_odh-ai-gateway-infra(012a9e62-3609-49b1-8741-0f30c617c8ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" Nov 03 21:00:05.663586 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.663565 5815 generic.go:356] "Generic (PLEG): container finished" podID="53905733-a798-40db-8e03-7bfc5e0152a9" containerID="960447ff2acf674fe7a020a85ba35063dc15ce92f0352e8808c91d9064edb237" exitCode=60 Nov 03 21:00:05.663696 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.663646 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" event={"ID":"53905733-a798-40db-8e03-7bfc5e0152a9","Type":"ContainerDied","Data":"960447ff2acf674fe7a020a85ba35063dc15ce92f0352e8808c91d9064edb237"} Nov 03 21:00:05.663844 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.663829 5815 scope.go:122] "RemoveContainer" containerID="960447ff2acf674fe7a020a85ba35063dc15ce92f0352e8808c91d9064edb237" Nov 03 21:00:05.664057 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.664035 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr_odh-ai-gateway-infra(53905733-a798-40db-8e03-7bfc5e0152a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" Nov 03 21:00:05.665653 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.665625 5815 generic.go:356] "Generic (PLEG): container finished" podID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerID="96644205d254553b9d903d9d5359898228278b1b0bcf71acb627d469000c6071" exitCode=60 Nov 03 21:00:05.665771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.665750 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" event={"ID":"82750f5b-62f3-4282-9ca1-59613ffb45d8","Type":"ContainerDied","Data":"96644205d254553b9d903d9d5359898228278b1b0bcf71acb627d469000c6071"} Nov 03 21:00:05.666181 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.665948 5815 scope.go:122] "RemoveContainer" containerID="96644205d254553b9d903d9d5359898228278b1b0bcf71acb627d469000c6071" Nov 03 21:00:05.666181 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.666143 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk_odh-ai-gateway-infra(82750f5b-62f3-4282-9ca1-59613ffb45d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" Nov 03 21:00:05.667908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.667674 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" event={"ID":"d7781320-da95-4c12-9645-b22679a15067","Type":"ContainerDied","Data":"ed1ca26004bd4d68e0a41a918806d12cf55fdbfcbff05fe33af835de41b1625f"} Nov 03 21:00:05.667908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.667763 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn" Nov 03 21:00:05.670131 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.670110 5815 generic.go:356] "Generic (PLEG): container finished" podID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerID="393210eccae4c7e209d4a1492452c79eacf062aee175f1bda4aa18455cd050a2" exitCode=60 Nov 03 21:00:05.670226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.670198 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" event={"ID":"f1bcb275-c36a-4919-9526-afe5f86e8ad3","Type":"ContainerDied","Data":"393210eccae4c7e209d4a1492452c79eacf062aee175f1bda4aa18455cd050a2"} Nov 03 21:00:05.670450 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.670435 5815 scope.go:122] "RemoveContainer" containerID="393210eccae4c7e209d4a1492452c79eacf062aee175f1bda4aa18455cd050a2" Nov 03 21:00:05.670662 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.670643 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5_odh-ai-gateway-infra(f1bcb275-c36a-4919-9526-afe5f86e8ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" Nov 03 21:00:05.674214 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.673727 5815 generic.go:356] "Generic (PLEG): container finished" podID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerID="27fdb079b279b2fcb441e3f64d9687d416746ba9c8c8b65a1b5e46e8b612610f" exitCode=60 Nov 03 21:00:05.674214 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.673821 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" event={"ID":"191ab3e9-ce49-4655-830a-7dc8160e9749","Type":"ContainerDied","Data":"27fdb079b279b2fcb441e3f64d9687d416746ba9c8c8b65a1b5e46e8b612610f"} Nov 03 21:00:05.674214 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.673998 5815 scope.go:122] "RemoveContainer" containerID="27fdb079b279b2fcb441e3f64d9687d416746ba9c8c8b65a1b5e46e8b612610f" Nov 03 21:00:05.674436 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.674222 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg_odh-ai-gateway-infra(191ab3e9-ce49-4655-830a-7dc8160e9749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" Nov 03 21:00:05.676844 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.676657 5815 generic.go:356] "Generic (PLEG): container finished" podID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerID="84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c" exitCode=60 Nov 03 21:00:05.676844 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.676767 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerDied","Data":"84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c"} Nov 03 21:00:05.677270 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.677250 5815 scope.go:122] "RemoveContainer" containerID="84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c" Nov 03 21:00:05.677540 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.677496 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc_odh-ai-gateway-infra(ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" Nov 03 21:00:05.678871 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.678849 5815 generic.go:356] "Generic (PLEG): container finished" podID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerID="dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b" exitCode=60 Nov 03 21:00:05.678964 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.678922 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerDied","Data":"dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b"} Nov 03 21:00:05.679095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.679080 5815 scope.go:122] "RemoveContainer" containerID="dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b" Nov 03 21:00:05.679291 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.679268 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp_odh-ai-gateway-infra(b0050a22-2ff2-48ac-be21-1fa149bcf98a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" Nov 03 21:00:05.681043 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.681022 5815 generic.go:356] "Generic (PLEG): container finished" podID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerID="611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01" exitCode=60 Nov 03 21:00:05.681148 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.681076 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerDied","Data":"611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01"} Nov 03 21:00:05.681398 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.681368 5815 scope.go:122] "RemoveContainer" containerID="611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01" Nov 03 21:00:05.681733 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.681704 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x_odh-ai-gateway-infra(7da0af6e-dcdc-4120-a6f7-85aea41453c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" Nov 03 21:00:05.683050 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.683032 5815 generic.go:356] "Generic (PLEG): container finished" podID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerID="cd4a5170ba7be0ad0ec0f88d4c54e5bdfa2b97b4e473c778139cfac2a2e2352b" exitCode=60 Nov 03 21:00:05.683123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.683069 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" event={"ID":"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e","Type":"ContainerDied","Data":"cd4a5170ba7be0ad0ec0f88d4c54e5bdfa2b97b4e473c778139cfac2a2e2352b"} Nov 03 21:00:05.683314 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.683302 5815 scope.go:122] "RemoveContainer" containerID="cd4a5170ba7be0ad0ec0f88d4c54e5bdfa2b97b4e473c778139cfac2a2e2352b" Nov 03 21:00:05.683477 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:05.683462 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw_odh-ai-gateway-infra(6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" Nov 03 21:00:05.700481 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.700453 5815 scope.go:122] "RemoveContainer" containerID="fa45f975360134dc0d7ea9289bbb5f801cd406a227d8f233df7f55a9a9299e47" Nov 03 21:00:05.716358 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.716338 5815 scope.go:122] "RemoveContainer" containerID="df02b73335038ca3e405fdcc4c7becdfd90d44a1be7e1b4f1f0afd3086d1b047" Nov 03 21:00:05.738016 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.736115 5815 scope.go:122] "RemoveContainer" containerID="cee4b17de921cd9f47a1e68f4be82d784dd02f90cd4a75ccbbc7dd20b37a3589" Nov 03 21:00:05.752294 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.752274 5815 scope.go:122] "RemoveContainer" containerID="6ba19fc6197a89656f8707a3c37de0b06ac4973d68593fedaae2d7434e2e19b5" Nov 03 21:00:05.763357 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.763325 5815 scope.go:122] "RemoveContainer" containerID="8b474e782fe864cc161ac25198c447bc627a0db5d66dc0da55961a4e4de5f5fb" Nov 03 21:00:05.786592 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.786573 5815 scope.go:122] "RemoveContainer" containerID="0cd081e68d1829313518e5c0fafb7be5b9b7bfc03cdc25519d185df38b73b24a" Nov 03 21:00:05.805990 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.805968 5815 scope.go:122] "RemoveContainer" containerID="bbb3878dc6ae8be7ad3526767a7eb99b0b97aaacd33ae0044f730b5bfb9d68c1" Nov 03 21:00:05.821125 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.821103 5815 scope.go:122] "RemoveContainer" containerID="5065a2220f223807aa45a460cea2283ce7003ce6dca2b46153774286d908801d" Nov 03 21:00:05.835558 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.835538 5815 scope.go:122] "RemoveContainer" containerID="3caa77faedd47ce304cf81db691e43247416c42a6dc3d25cb6daf60ede9d251c" Nov 03 21:00:05.864849 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.864794 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns"] Nov 03 21:00:05.894285 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.894224 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/premium-w4-466f83-kserve-5d655b97bc-dmpns"] Nov 03 21:00:05.894756 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.894731 5815 scope.go:122] "RemoveContainer" containerID="ce1ee7b7282c57fe53f203e87044bbf1027df2b8b888a0d6ea58f94c4de0dbe1" Nov 03 21:00:05.916289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.916270 5815 scope.go:122] "RemoveContainer" containerID="eb46131212b6d1ece5f2836face0fe9cc27f9b9c00fe1d5e67f156664b4a0686" Nov 03 21:00:05.931182 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.931164 5815 scope.go:122] "RemoveContainer" containerID="f86dddebaea884033331b715ffd77f6a5ba6035a7d7a0996a07bcb43c2c67076" Nov 03 21:00:05.946432 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.946400 5815 scope.go:122] "RemoveContainer" containerID="1d468ef5dddf876f6890dfd3f44abcf411824eb64c88520eb30ba74a542117ff" Nov 03 21:00:05.950675 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.950477 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn"] Nov 03 21:00:05.955107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.955086 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-2-w4-466f83-kserve-fb6896b4f-v2gzn"] Nov 03 21:00:05.960759 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.960736 5815 scope.go:122] "RemoveContainer" containerID="e7692f5f5d2916002a5a4d3a20ae8015e2374e5c9a7d865439e73d962f8e8538" Nov 03 21:00:05.983126 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:05.983104 5815 scope.go:122] "RemoveContainer" containerID="a90ac36f4eb3deae2e48c1d98abeac3cd80b7a24c6ecb40d6128cad2ff28a80e" Nov 03 21:00:06.590434 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.590398 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" path="/var/lib/kubelet/pods/baf361fa-7064-41ed-af8d-c674ae0eb7d1/volumes" Nov 03 21:00:06.590949 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.590933 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7781320-da95-4c12-9645-b22679a15067" path="/var/lib/kubelet/pods/d7781320-da95-4c12-9645-b22679a15067/volumes" Nov 03 21:00:06.696641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.696616 5815 scope.go:122] "RemoveContainer" containerID="27fdb079b279b2fcb441e3f64d9687d416746ba9c8c8b65a1b5e46e8b612610f" Nov 03 21:00:06.697036 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.696930 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg_odh-ai-gateway-infra(191ab3e9-ce49-4655-830a-7dc8160e9749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" Nov 03 21:00:06.699950 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.699927 5815 scope.go:122] "RemoveContainer" containerID="84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c" Nov 03 21:00:06.700232 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.700207 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc_odh-ai-gateway-infra(ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" Nov 03 21:00:06.701783 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.701758 5815 scope.go:122] "RemoveContainer" containerID="dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b" Nov 03 21:00:06.702033 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.702008 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp_odh-ai-gateway-infra(b0050a22-2ff2-48ac-be21-1fa149bcf98a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" Nov 03 21:00:06.703662 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.703643 5815 scope.go:122] "RemoveContainer" containerID="611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01" Nov 03 21:00:06.703974 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.703950 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x_odh-ai-gateway-infra(7da0af6e-dcdc-4120-a6f7-85aea41453c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" Nov 03 21:00:06.706152 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.706134 5815 scope.go:122] "RemoveContainer" containerID="cd4a5170ba7be0ad0ec0f88d4c54e5bdfa2b97b4e473c778139cfac2a2e2352b" Nov 03 21:00:06.706362 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.706337 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw_odh-ai-gateway-infra(6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" Nov 03 21:00:06.708088 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.708071 5815 scope.go:122] "RemoveContainer" containerID="de58eac52c99175d2fd5b13de90c89327f803e1623cfcaaa65189df99ec2743b" Nov 03 21:00:06.708317 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.708291 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g_odh-ai-gateway-infra(1863d0ed-1093-4ba8-adbe-bb0ce554ff8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" Nov 03 21:00:06.710050 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.710031 5815 scope.go:122] "RemoveContainer" containerID="922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f" Nov 03 21:00:06.710299 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.710266 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn_odh-ai-gateway-infra(c14c22cc-8192-4cc4-86a5-bd11e23de11d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" Nov 03 21:00:06.711981 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.711939 5815 scope.go:122] "RemoveContainer" containerID="776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334" Nov 03 21:00:06.712222 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.712196 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c_odh-ai-gateway-infra(93056b14-35ff-4299-ae03-ce994d5f31f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" Nov 03 21:00:06.713950 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.713930 5815 scope.go:122] "RemoveContainer" containerID="de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58" Nov 03 21:00:06.714203 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.714177 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs_odh-ai-gateway-infra(a16e4409-5714-49f4-9af0-c36e0760fcd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" Nov 03 21:00:06.716753 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.716736 5815 scope.go:122] "RemoveContainer" containerID="569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b" Nov 03 21:00:06.716963 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.716939 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f_odh-ai-gateway-infra(c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" Nov 03 21:00:06.719090 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.719073 5815 scope.go:122] "RemoveContainer" containerID="0540b8bb2657f14ae1228018ed97e235ae5b8c6b8ba3cd4ded5807c0a34b8d16" Nov 03 21:00:06.719321 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.719299 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2_odh-ai-gateway-infra(012a9e62-3609-49b1-8741-0f30c617c8ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" Nov 03 21:00:06.720788 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.720767 5815 scope.go:122] "RemoveContainer" containerID="960447ff2acf674fe7a020a85ba35063dc15ce92f0352e8808c91d9064edb237" Nov 03 21:00:06.721012 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.720992 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr_odh-ai-gateway-infra(53905733-a798-40db-8e03-7bfc5e0152a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" Nov 03 21:00:06.722393 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.722376 5815 scope.go:122] "RemoveContainer" containerID="96644205d254553b9d903d9d5359898228278b1b0bcf71acb627d469000c6071" Nov 03 21:00:06.722617 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.722596 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk_odh-ai-gateway-infra(82750f5b-62f3-4282-9ca1-59613ffb45d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" Nov 03 21:00:06.725198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:06.725183 5815 scope.go:122] "RemoveContainer" containerID="393210eccae4c7e209d4a1492452c79eacf062aee175f1bda4aa18455cd050a2" Nov 03 21:00:06.725373 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:06.725357 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5_odh-ai-gateway-infra(f1bcb275-c36a-4919-9526-afe5f86e8ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" Nov 03 21:00:07.189253 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:07.189218 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:00:11.242208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.242132 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg"] Nov 03 21:00:11.242883 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.242858 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" Nov 03 21:00:11.242883 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.242875 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" Nov 03 21:00:11.242985 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.242902 5815 cpu_manager.go:414] "containerMap: removing container" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" Nov 03 21:00:11.242985 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.242907 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" Nov 03 21:00:11.243077 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.243056 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7781320-da95-4c12-9645-b22679a15067" containerName="main" Nov 03 21:00:11.243077 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.243069 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf361fa-7064-41ed-af8d-c674ae0eb7d1" containerName="main" Nov 03 21:00:11.248129 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.248111 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.253348 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.253323 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:00:11.254996 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.254969 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg"] Nov 03 21:00:11.339454 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.339425 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.339630 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.339575 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.339692 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.339637 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdf5w\" (UniqueName: \"kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.440885 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.440860 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdf5w\" (UniqueName: \"kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.441046 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.440995 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.441092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.441072 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.441711 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.441679 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.443584 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.443566 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.454344 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.454325 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdf5w\" (UniqueName: \"kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w\") pod \"maas-api-revoke-sub-38df0deb-92xsg\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.578402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.578343 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:11.865435 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:11.865405 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg"] Nov 03 21:00:11.865944 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:11.865901 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4001712_c784_4ec4_ba7e_37ba30678285.slice/crio-06b3a5c44e15b648279e92beffae30ebedd81918bdffcc39c19ca2dde7584ee4 WatchSource:0}: Error finding container 06b3a5c44e15b648279e92beffae30ebedd81918bdffcc39c19ca2dde7584ee4: Status 404 returned error can't find the container with id 06b3a5c44e15b648279e92beffae30ebedd81918bdffcc39c19ca2dde7584ee4 Nov 03 21:00:12.834801 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:12.834758 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" event={"ID":"e4001712-c784-4ec4-ba7e-37ba30678285","Type":"ContainerStarted","Data":"20d8952a444b51e8ece710e2605765cd17263c2bfc9bd25fba64acf265ebced9"} Nov 03 21:00:12.835205 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:12.834823 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" event={"ID":"e4001712-c784-4ec4-ba7e-37ba30678285","Type":"ContainerStarted","Data":"06b3a5c44e15b648279e92beffae30ebedd81918bdffcc39c19ca2dde7584ee4"} Nov 03 21:00:12.855516 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:12.855452 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" podStartSLOduration=1.855440397 podStartE2EDuration="1.855440397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:12.852423829 +0000 UTC m=+3522.382769500" watchObservedRunningTime="2026-11-03 21:00:12.855440397 +0000 UTC m=+3522.385786069" Nov 03 21:00:13.861562 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:13.861496 5815 generic.go:356] "Generic (PLEG): container finished" podID="e4001712-c784-4ec4-ba7e-37ba30678285" containerID="20d8952a444b51e8ece710e2605765cd17263c2bfc9bd25fba64acf265ebced9" exitCode=0 Nov 03 21:00:13.862428 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:13.862397 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" event={"ID":"e4001712-c784-4ec4-ba7e-37ba30678285","Type":"ContainerDied","Data":"20d8952a444b51e8ece710e2605765cd17263c2bfc9bd25fba64acf265ebced9"} Nov 03 21:00:13.903877 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:13.903846 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:00:15.050888 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.050867 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:15.102372 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.102336 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca\") pod \"e4001712-c784-4ec4-ba7e-37ba30678285\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " Nov 03 21:00:15.102618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.102473 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca\") pod \"e4001712-c784-4ec4-ba7e-37ba30678285\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " Nov 03 21:00:15.102618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.102585 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w\" (UniqueName: \"kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w\") pod \"e4001712-c784-4ec4-ba7e-37ba30678285\" (UID: \"e4001712-c784-4ec4-ba7e-37ba30678285\") " Nov 03 21:00:15.103082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.103053 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca" pod "e4001712-c784-4ec4-ba7e-37ba30678285" (UID: "e4001712-c784-4ec4-ba7e-37ba30678285"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:15.103266 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.103247 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:15.105699 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.105675 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca" pod "e4001712-c784-4ec4-ba7e-37ba30678285" (UID: "e4001712-c784-4ec4-ba7e-37ba30678285"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:15.105931 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.105912 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w" pod "e4001712-c784-4ec4-ba7e-37ba30678285" (UID: "e4001712-c784-4ec4-ba7e-37ba30678285"). InnerVolumeSpecName "kube-api-access-mdf5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:15.205123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.205060 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdf5w\" (UniqueName: \"kubernetes.io/projected/e4001712-c784-4ec4-ba7e-37ba30678285-kube-api-access-mdf5w\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:15.205123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.205085 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4001712-c784-4ec4-ba7e-37ba30678285-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:15.578736 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.578710 5815 scope.go:122] "RemoveContainer" containerID="776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334" Nov 03 21:00:15.578928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.578909 5815 scope.go:122] "RemoveContainer" containerID="569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b" Nov 03 21:00:15.882651 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.882562 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" Nov 03 21:00:15.882831 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.882573 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg" event={"ID":"e4001712-c784-4ec4-ba7e-37ba30678285","Type":"ContainerDied","Data":"06b3a5c44e15b648279e92beffae30ebedd81918bdffcc39c19ca2dde7584ee4"} Nov 03 21:00:15.882831 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:15.882716 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06b3a5c44e15b648279e92beffae30ebedd81918bdffcc39c19ca2dde7584ee4" Nov 03 21:00:16.578178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.578154 5815 scope.go:122] "RemoveContainer" containerID="84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c" Nov 03 21:00:16.578656 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.578305 5815 scope.go:122] "RemoveContainer" containerID="de58eac52c99175d2fd5b13de90c89327f803e1623cfcaaa65189df99ec2743b" Nov 03 21:00:16.952385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.952347 5815 generic.go:356] "Generic (PLEG): container finished" podID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerID="c0897b46dceb038da536f90e1f84cca174bf1aa3fb6ced9fe4e71fb7acb9770b" exitCode=60 Nov 03 21:00:16.952595 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.952423 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" event={"ID":"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5","Type":"ContainerDied","Data":"c0897b46dceb038da536f90e1f84cca174bf1aa3fb6ced9fe4e71fb7acb9770b"} Nov 03 21:00:16.952595 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.952497 5815 scope.go:122] "RemoveContainer" containerID="569c548b52c6a7fd8eb151316642058ec9a3bcd23c59060625debbe3268f4f4b" Nov 03 21:00:16.953425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.953401 5815 scope.go:122] "RemoveContainer" containerID="c0897b46dceb038da536f90e1f84cca174bf1aa3fb6ced9fe4e71fb7acb9770b" Nov 03 21:00:16.953796 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:16.953771 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f_odh-ai-gateway-infra(c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" Nov 03 21:00:16.955724 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.955699 5815 generic.go:356] "Generic (PLEG): container finished" podID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerID="a1566e377a834e9fbe213f5eb609e67ab9677c51cee1a2e6bc1762f8a3e147da" exitCode=60 Nov 03 21:00:16.955855 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.955763 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerDied","Data":"a1566e377a834e9fbe213f5eb609e67ab9677c51cee1a2e6bc1762f8a3e147da"} Nov 03 21:00:16.956130 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.956104 5815 scope.go:122] "RemoveContainer" containerID="a1566e377a834e9fbe213f5eb609e67ab9677c51cee1a2e6bc1762f8a3e147da" Nov 03 21:00:16.957069 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:16.957020 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c_odh-ai-gateway-infra(93056b14-35ff-4299-ae03-ce994d5f31f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" Nov 03 21:00:16.978541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:16.978360 5815 scope.go:122] "RemoveContainer" containerID="776f6a4dc7b2b40684caec9084e9ab70115a370907233c8edc9ca16c40cdc334" Nov 03 21:00:17.579560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.578144 5815 scope.go:122] "RemoveContainer" containerID="0540b8bb2657f14ae1228018ed97e235ae5b8c6b8ba3cd4ded5807c0a34b8d16" Nov 03 21:00:17.579560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.578779 5815 scope.go:122] "RemoveContainer" containerID="611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01" Nov 03 21:00:17.579560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.578872 5815 scope.go:122] "RemoveContainer" containerID="27fdb079b279b2fcb441e3f64d9687d416746ba9c8c8b65a1b5e46e8b612610f" Nov 03 21:00:17.579560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.579221 5815 scope.go:122] "RemoveContainer" containerID="96644205d254553b9d903d9d5359898228278b1b0bcf71acb627d469000c6071" Nov 03 21:00:17.579560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.579356 5815 scope.go:122] "RemoveContainer" containerID="dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b" Nov 03 21:00:17.980592 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.980556 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerDied","Data":"b3f97188e5bd781ade910c9a328d7d853007d4089637d3ceb936a0feb325c9ab"} Nov 03 21:00:17.980760 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.980638 5815 scope.go:122] "RemoveContainer" containerID="84b6fb5e50c622fa60e755ef8ec18640f963499a4fff4d2e09abd7c9ee02d37c" Nov 03 21:00:17.980760 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.980563 5815 generic.go:356] "Generic (PLEG): container finished" podID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerID="b3f97188e5bd781ade910c9a328d7d853007d4089637d3ceb936a0feb325c9ab" exitCode=60 Nov 03 21:00:17.981121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.981099 5815 scope.go:122] "RemoveContainer" containerID="b3f97188e5bd781ade910c9a328d7d853007d4089637d3ceb936a0feb325c9ab" Nov 03 21:00:17.981536 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:17.981494 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc_odh-ai-gateway-infra(ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" Nov 03 21:00:17.986359 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.986265 5815 generic.go:356] "Generic (PLEG): container finished" podID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerID="c28a5057ce43e15fd3bb4f44a4d2b3a9e8f8e525d1121d840f1f7d53752bc810" exitCode=60 Nov 03 21:00:17.986359 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.986342 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" event={"ID":"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e","Type":"ContainerDied","Data":"c28a5057ce43e15fd3bb4f44a4d2b3a9e8f8e525d1121d840f1f7d53752bc810"} Nov 03 21:00:17.987125 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.986776 5815 scope.go:122] "RemoveContainer" containerID="c28a5057ce43e15fd3bb4f44a4d2b3a9e8f8e525d1121d840f1f7d53752bc810" Nov 03 21:00:17.987125 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:17.987082 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g_odh-ai-gateway-infra(1863d0ed-1093-4ba8-adbe-bb0ce554ff8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" Nov 03 21:00:17.997563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:17.997535 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f"] Nov 03 21:00:18.010302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.010276 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c"] Nov 03 21:00:18.010763 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.010697 5815 scope.go:122] "RemoveContainer" containerID="de58eac52c99175d2fd5b13de90c89327f803e1623cfcaaa65189df99ec2743b" Nov 03 21:00:18.207110 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.207080 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:18.251360 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.251334 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:18.282274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.272001 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca\") pod \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " Nov 03 21:00:18.282695 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.282662 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m\" (UniqueName: \"kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m\") pod \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\" (UID: \"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5\") " Nov 03 21:00:18.282827 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.272898 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca" pod "c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" (UID: "c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.283582 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.283561 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.290094 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.290056 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m" pod "c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" (UID: "c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5"). InnerVolumeSpecName "kube-api-access-rds6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.384372 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.384338 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca\") pod \"93056b14-35ff-4299-ae03-ce994d5f31f3\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " Nov 03 21:00:18.384603 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.384424 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z\" (UniqueName: \"kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z\") pod \"93056b14-35ff-4299-ae03-ce994d5f31f3\" (UID: \"93056b14-35ff-4299-ae03-ce994d5f31f3\") " Nov 03 21:00:18.385105 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.385082 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rds6m\" (UniqueName: \"kubernetes.io/projected/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5-kube-api-access-rds6m\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.385719 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.385691 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca" pod "93056b14-35ff-4299-ae03-ce994d5f31f3" (UID: "93056b14-35ff-4299-ae03-ce994d5f31f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.387710 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.387680 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z" pod "93056b14-35ff-4299-ae03-ce994d5f31f3" (UID: "93056b14-35ff-4299-ae03-ce994d5f31f3"). InnerVolumeSpecName "kube-api-access-cq75z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.486463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.486425 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93056b14-35ff-4299-ae03-ce994d5f31f3-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.486463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.486465 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq75z\" (UniqueName: \"kubernetes.io/projected/93056b14-35ff-4299-ae03-ce994d5f31f3-kube-api-access-cq75z\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.579000 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.578969 5815 scope.go:122] "RemoveContainer" containerID="cd4a5170ba7be0ad0ec0f88d4c54e5bdfa2b97b4e473c778139cfac2a2e2352b" Nov 03 21:00:18.579191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.579131 5815 scope.go:122] "RemoveContainer" containerID="922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f" Nov 03 21:00:18.580457 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.580423 5815 scope.go:122] "RemoveContainer" containerID="393210eccae4c7e209d4a1492452c79eacf062aee175f1bda4aa18455cd050a2" Nov 03 21:00:18.580972 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:18.580896 5815 scope.go:122] "RemoveContainer" containerID="960447ff2acf674fe7a020a85ba35063dc15ce92f0352e8808c91d9064edb237" Nov 03 21:00:19.011348 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.011321 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" event={"ID":"c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5","Type":"ContainerDied","Data":"8b52d02b1b81beaa99b37a8d2d7051c98f08011ed415fa08eec7c7fa345f6662"} Nov 03 21:00:19.011491 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.011326 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f" Nov 03 21:00:19.011491 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.011388 5815 scope.go:122] "RemoveContainer" containerID="c0897b46dceb038da536f90e1f84cca174bf1aa3fb6ced9fe4e71fb7acb9770b" Nov 03 21:00:19.015817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.015779 5815 generic.go:356] "Generic (PLEG): container finished" podID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerID="043084d4ca037752dd998c2fc85be7983ba8f4ff27ee8d959e9ff2c381932a6c" exitCode=60 Nov 03 21:00:19.015962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.015860 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" event={"ID":"012a9e62-3609-49b1-8741-0f30c617c8ea","Type":"ContainerDied","Data":"043084d4ca037752dd998c2fc85be7983ba8f4ff27ee8d959e9ff2c381932a6c"} Nov 03 21:00:19.016325 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.016305 5815 scope.go:122] "RemoveContainer" containerID="043084d4ca037752dd998c2fc85be7983ba8f4ff27ee8d959e9ff2c381932a6c" Nov 03 21:00:19.016694 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:19.016667 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2_odh-ai-gateway-infra(012a9e62-3609-49b1-8741-0f30c617c8ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" Nov 03 21:00:19.018949 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.018925 5815 generic.go:356] "Generic (PLEG): container finished" podID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerID="e1923eb4afc87cc8c174eb5f14838472e329d055633034af0bb845d226318b8d" exitCode=60 Nov 03 21:00:19.019035 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.018996 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" event={"ID":"82750f5b-62f3-4282-9ca1-59613ffb45d8","Type":"ContainerDied","Data":"e1923eb4afc87cc8c174eb5f14838472e329d055633034af0bb845d226318b8d"} Nov 03 21:00:19.019212 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.019197 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc"] Nov 03 21:00:19.019360 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.019346 5815 scope.go:122] "RemoveContainer" containerID="e1923eb4afc87cc8c174eb5f14838472e329d055633034af0bb845d226318b8d" Nov 03 21:00:19.019661 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:19.019627 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk_odh-ai-gateway-infra(82750f5b-62f3-4282-9ca1-59613ffb45d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" Nov 03 21:00:19.021089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.021042 5815 generic.go:356] "Generic (PLEG): container finished" podID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerID="26ca7239fdd9e010f1c77b5323ac9888d8521f2d02380cb6e3312baf8de9e883" exitCode=60 Nov 03 21:00:19.021192 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.021112 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" event={"ID":"191ab3e9-ce49-4655-830a-7dc8160e9749","Type":"ContainerDied","Data":"26ca7239fdd9e010f1c77b5323ac9888d8521f2d02380cb6e3312baf8de9e883"} Nov 03 21:00:19.021547 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.021494 5815 scope.go:122] "RemoveContainer" containerID="26ca7239fdd9e010f1c77b5323ac9888d8521f2d02380cb6e3312baf8de9e883" Nov 03 21:00:19.021790 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:19.021772 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg_odh-ai-gateway-infra(191ab3e9-ce49-4655-830a-7dc8160e9749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" Nov 03 21:00:19.025178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.025151 5815 generic.go:356] "Generic (PLEG): container finished" podID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerID="e46e58ef1c5dc5546cfa75fa704b1147641780439412a9f03c30ac6eba50ecd8" exitCode=60 Nov 03 21:00:19.025298 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.025223 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerDied","Data":"e46e58ef1c5dc5546cfa75fa704b1147641780439412a9f03c30ac6eba50ecd8"} Nov 03 21:00:19.025624 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.025602 5815 scope.go:122] "RemoveContainer" containerID="e46e58ef1c5dc5546cfa75fa704b1147641780439412a9f03c30ac6eba50ecd8" Nov 03 21:00:19.025905 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:19.025885 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp_odh-ai-gateway-infra(b0050a22-2ff2-48ac-be21-1fa149bcf98a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" Nov 03 21:00:19.027427 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.027406 5815 generic.go:356] "Generic (PLEG): container finished" podID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerID="2ee399ce1de21abe777a5d2646deb79cbb1ab951554c323a43895bfe576764fc" exitCode=60 Nov 03 21:00:19.027551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.027476 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerDied","Data":"2ee399ce1de21abe777a5d2646deb79cbb1ab951554c323a43895bfe576764fc"} Nov 03 21:00:19.027977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.027956 5815 scope.go:122] "RemoveContainer" containerID="2ee399ce1de21abe777a5d2646deb79cbb1ab951554c323a43895bfe576764fc" Nov 03 21:00:19.028270 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:19.028246 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x_odh-ai-gateway-infra(7da0af6e-dcdc-4120-a6f7-85aea41453c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" Nov 03 21:00:19.031061 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.031039 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" event={"ID":"93056b14-35ff-4299-ae03-ce994d5f31f3","Type":"ContainerDied","Data":"de33e130874eb778fbf8f111d28af737da947612d0aa63b96bfea56a3694e306"} Nov 03 21:00:19.031180 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.031064 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c" Nov 03 21:00:19.033584 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.033562 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g"] Nov 03 21:00:19.127815 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.127780 5815 scope.go:122] "RemoveContainer" containerID="0540b8bb2657f14ae1228018ed97e235ae5b8c6b8ba3cd4ded5807c0a34b8d16" Nov 03 21:00:19.159038 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.159013 5815 scope.go:122] "RemoveContainer" containerID="96644205d254553b9d903d9d5359898228278b1b0bcf71acb627d469000c6071" Nov 03 21:00:19.228406 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.228372 5815 scope.go:122] "RemoveContainer" containerID="27fdb079b279b2fcb441e3f64d9687d416746ba9c8c8b65a1b5e46e8b612610f" Nov 03 21:00:19.248984 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.248961 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:19.261720 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.261683 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f"] Nov 03 21:00:19.274121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.274044 5815 scope.go:122] "RemoveContainer" containerID="dba7c42aebd541fee518774958cb48e790469b8af66e1167fdb8c16526ffd08b" Nov 03 21:00:19.283672 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.283601 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895660-wh49f"] Nov 03 21:00:19.295465 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.295410 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c"] Nov 03 21:00:19.302965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.302939 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bab2c3-29895660-wrr7c"] Nov 03 21:00:19.317818 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.313995 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca\") pod \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " Nov 03 21:00:19.317818 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.314103 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp\" (UniqueName: \"kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp\") pod \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\" (UID: \"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6\") " Nov 03 21:00:19.317818 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.317270 5815 scope.go:122] "RemoveContainer" containerID="611f9599adab3b280b3bef8503459059c4c701cba39cad85628847ce366aac01" Nov 03 21:00:19.318309 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.318168 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca" pod "ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" (UID: "ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:19.321033 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.318963 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp" pod "ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" (UID: "ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6"). InnerVolumeSpecName "kube-api-access-jhtgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:19.326848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.326489 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:19.336360 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.336330 5815 scope.go:122] "RemoveContainer" containerID="a1566e377a834e9fbe213f5eb609e67ab9677c51cee1a2e6bc1762f8a3e147da" Nov 03 21:00:19.416193 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.416162 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca\") pod \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " Nov 03 21:00:19.416347 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.416227 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc\" (UniqueName: \"kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc\") pod \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\" (UID: \"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e\") " Nov 03 21:00:19.416627 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.416602 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca" pod "1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" (UID: "1863d0ed-1093-4ba8-adbe-bb0ce554ff8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:19.417090 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.417067 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.417090 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.417090 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.417212 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.417100 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtgp\" (UniqueName: \"kubernetes.io/projected/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6-kube-api-access-jhtgp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.418616 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.418593 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc" pod "1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" (UID: "1863d0ed-1093-4ba8-adbe-bb0ce554ff8e"). InnerVolumeSpecName "kube-api-access-q4xhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:19.525039 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.524950 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4xhc\" (UniqueName: \"kubernetes.io/projected/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e-kube-api-access-q4xhc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.578904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:19.578860 5815 scope.go:122] "RemoveContainer" containerID="de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58" Nov 03 21:00:20.043726 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.043697 5815 generic.go:356] "Generic (PLEG): container finished" podID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerID="440b51521e9ed0a4c0687d52cb74f4140c7934d35cc1d3649c203824a25d2929" exitCode=60 Nov 03 21:00:20.044116 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.043768 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" event={"ID":"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e","Type":"ContainerDied","Data":"440b51521e9ed0a4c0687d52cb74f4140c7934d35cc1d3649c203824a25d2929"} Nov 03 21:00:20.044116 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.043832 5815 scope.go:122] "RemoveContainer" containerID="cd4a5170ba7be0ad0ec0f88d4c54e5bdfa2b97b4e473c778139cfac2a2e2352b" Nov 03 21:00:20.044291 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.044274 5815 scope.go:122] "RemoveContainer" containerID="440b51521e9ed0a4c0687d52cb74f4140c7934d35cc1d3649c203824a25d2929" Nov 03 21:00:20.044646 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:20.044621 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw_odh-ai-gateway-infra(6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" Nov 03 21:00:20.045414 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.045399 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" Nov 03 21:00:20.045486 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.045416 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g" event={"ID":"1863d0ed-1093-4ba8-adbe-bb0ce554ff8e","Type":"ContainerDied","Data":"2d69e30d4c3ca2816393f8a48d9414df34548e5be138f2200e8aa5d3e64857b3"} Nov 03 21:00:20.047562 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.047540 5815 generic.go:356] "Generic (PLEG): container finished" podID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerID="82bab08a409a4c3d71ecbfbd40017f70d0bf6cfe76c945907ae59d680dfe3303" exitCode=60 Nov 03 21:00:20.047655 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.047547 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerDied","Data":"82bab08a409a4c3d71ecbfbd40017f70d0bf6cfe76c945907ae59d680dfe3303"} Nov 03 21:00:20.047992 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.047970 5815 scope.go:122] "RemoveContainer" containerID="82bab08a409a4c3d71ecbfbd40017f70d0bf6cfe76c945907ae59d680dfe3303" Nov 03 21:00:20.048728 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:20.048357 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn_odh-ai-gateway-infra(c14c22cc-8192-4cc4-86a5-bd11e23de11d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" Nov 03 21:00:20.054878 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.054859 5815 generic.go:356] "Generic (PLEG): container finished" podID="53905733-a798-40db-8e03-7bfc5e0152a9" containerID="72a20649a3b1357fda62ed9ceb5bd2370d2e058f5fc7d6720ff601be81729fe1" exitCode=60 Nov 03 21:00:20.054979 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.054932 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" event={"ID":"53905733-a798-40db-8e03-7bfc5e0152a9","Type":"ContainerDied","Data":"72a20649a3b1357fda62ed9ceb5bd2370d2e058f5fc7d6720ff601be81729fe1"} Nov 03 21:00:20.055251 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.055235 5815 scope.go:122] "RemoveContainer" containerID="72a20649a3b1357fda62ed9ceb5bd2370d2e058f5fc7d6720ff601be81729fe1" Nov 03 21:00:20.055382 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.055360 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2"] Nov 03 21:00:20.055470 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:20.055452 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr_odh-ai-gateway-infra(53905733-a798-40db-8e03-7bfc5e0152a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" Nov 03 21:00:20.059346 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.059328 5815 generic.go:356] "Generic (PLEG): container finished" podID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerID="535923c3bd01dce45fb6fa4c3ed14141a2cca476882f2f637f63921dc7f7b430" exitCode=60 Nov 03 21:00:20.059442 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.059393 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" event={"ID":"f1bcb275-c36a-4919-9526-afe5f86e8ad3","Type":"ContainerDied","Data":"535923c3bd01dce45fb6fa4c3ed14141a2cca476882f2f637f63921dc7f7b430"} Nov 03 21:00:20.059702 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.059686 5815 scope.go:122] "RemoveContainer" containerID="535923c3bd01dce45fb6fa4c3ed14141a2cca476882f2f637f63921dc7f7b430" Nov 03 21:00:20.059944 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:20.059923 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5_odh-ai-gateway-infra(f1bcb275-c36a-4919-9526-afe5f86e8ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" Nov 03 21:00:20.062383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.062369 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" Nov 03 21:00:20.062481 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.062399 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc" event={"ID":"ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6","Type":"ContainerDied","Data":"ccbf0de2702d094fe7f03dcfd30d9fd91b4ee9a9d28c495417029f21fd819fdb"} Nov 03 21:00:20.083396 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.082444 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk"] Nov 03 21:00:20.102805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.102777 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x"] Nov 03 21:00:20.120119 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.120095 5815 scope.go:122] "RemoveContainer" containerID="c28a5057ce43e15fd3bb4f44a4d2b3a9e8f8e525d1121d840f1f7d53752bc810" Nov 03 21:00:20.135274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.135025 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg"] Nov 03 21:00:20.151599 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.148687 5815 scope.go:122] "RemoveContainer" containerID="922acb73d624c5f47a5d7946034b96f263b4d332ff2ad65885c5ab3600d7539f" Nov 03 21:00:20.216981 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.216949 5815 scope.go:122] "RemoveContainer" containerID="960447ff2acf674fe7a020a85ba35063dc15ce92f0352e8808c91d9064edb237" Nov 03 21:00:20.217852 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.217637 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp"] Nov 03 21:00:20.249849 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.249826 5815 scope.go:122] "RemoveContainer" containerID="393210eccae4c7e209d4a1492452c79eacf062aee175f1bda4aa18455cd050a2" Nov 03 21:00:20.277328 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.276177 5815 scope.go:122] "RemoveContainer" containerID="b3f97188e5bd781ade910c9a328d7d853007d4089637d3ceb936a0feb325c9ab" Nov 03 21:00:20.280951 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.280911 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g"] Nov 03 21:00:20.287805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.287745 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:20.289582 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.288973 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-db5198-29895660-rc49g"] Nov 03 21:00:20.297305 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.296795 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc"] Nov 03 21:00:20.302377 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.302357 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c7aecb-29895660-hh6kc"] Nov 03 21:00:20.345787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.342977 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca\") pod \"012a9e62-3609-49b1-8741-0f30c617c8ea\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " Nov 03 21:00:20.345787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.343089 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5\" (UniqueName: \"kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5\") pod \"012a9e62-3609-49b1-8741-0f30c617c8ea\" (UID: \"012a9e62-3609-49b1-8741-0f30c617c8ea\") " Nov 03 21:00:20.345787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.343368 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca" pod "012a9e62-3609-49b1-8741-0f30c617c8ea" (UID: "012a9e62-3609-49b1-8741-0f30c617c8ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.345787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.343718 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/012a9e62-3609-49b1-8741-0f30c617c8ea-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.351525 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.347018 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5" pod "012a9e62-3609-49b1-8741-0f30c617c8ea" (UID: "012a9e62-3609-49b1-8741-0f30c617c8ea"). InnerVolumeSpecName "kube-api-access-np6k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.454921 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.454875 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np6k5\" (UniqueName: \"kubernetes.io/projected/012a9e62-3609-49b1-8741-0f30c617c8ea-kube-api-access-np6k5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.517848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.497996 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:20.517848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.516202 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:20.519308 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.519231 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:20.556291 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.556200 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca\") pod \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " Nov 03 21:00:20.556457 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.556420 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26\" (UniqueName: \"kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26\") pod \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\" (UID: \"7da0af6e-dcdc-4120-a6f7-85aea41453c5\") " Nov 03 21:00:20.557270 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.557243 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca" pod "7da0af6e-dcdc-4120-a6f7-85aea41453c5" (UID: "7da0af6e-dcdc-4120-a6f7-85aea41453c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.567542 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.567479 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26" pod "7da0af6e-dcdc-4120-a6f7-85aea41453c5" (UID: "7da0af6e-dcdc-4120-a6f7-85aea41453c5"). InnerVolumeSpecName "kube-api-access-zgk26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.595431 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.595399 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" path="/var/lib/kubelet/pods/1863d0ed-1093-4ba8-adbe-bb0ce554ff8e/volumes" Nov 03 21:00:20.595974 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.595953 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:20.596107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.596086 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" path="/var/lib/kubelet/pods/93056b14-35ff-4299-ae03-ce994d5f31f3/volumes" Nov 03 21:00:20.596692 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.596671 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" path="/var/lib/kubelet/pods/ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6/volumes" Nov 03 21:00:20.602256 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.602215 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" path="/var/lib/kubelet/pods/c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5/volumes" Nov 03 21:00:20.658132 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.658103 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt\" (UniqueName: \"kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt\") pod \"82750f5b-62f3-4282-9ca1-59613ffb45d8\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " Nov 03 21:00:20.658263 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.658217 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc\" (UniqueName: \"kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc\") pod \"191ab3e9-ce49-4655-830a-7dc8160e9749\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " Nov 03 21:00:20.658461 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.658409 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca\") pod \"82750f5b-62f3-4282-9ca1-59613ffb45d8\" (UID: \"82750f5b-62f3-4282-9ca1-59613ffb45d8\") " Nov 03 21:00:20.658551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.658494 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca\") pod \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " Nov 03 21:00:20.658642 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.658619 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9\" (UniqueName: \"kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9\") pod \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\" (UID: \"b0050a22-2ff2-48ac-be21-1fa149bcf98a\") " Nov 03 21:00:20.658780 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.658655 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca\") pod \"191ab3e9-ce49-4655-830a-7dc8160e9749\" (UID: \"191ab3e9-ce49-4655-830a-7dc8160e9749\") " Nov 03 21:00:20.659116 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.659087 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca" pod "82750f5b-62f3-4282-9ca1-59613ffb45d8" (UID: "82750f5b-62f3-4282-9ca1-59613ffb45d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.659243 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.659212 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgk26\" (UniqueName: \"kubernetes.io/projected/7da0af6e-dcdc-4120-a6f7-85aea41453c5-kube-api-access-zgk26\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.659458 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.659402 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7da0af6e-dcdc-4120-a6f7-85aea41453c5-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.659647 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.659624 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca" pod "191ab3e9-ce49-4655-830a-7dc8160e9749" (UID: "191ab3e9-ce49-4655-830a-7dc8160e9749"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.659976 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.659958 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca" pod "b0050a22-2ff2-48ac-be21-1fa149bcf98a" (UID: "b0050a22-2ff2-48ac-be21-1fa149bcf98a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.661248 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.661222 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt" pod "82750f5b-62f3-4282-9ca1-59613ffb45d8" (UID: "82750f5b-62f3-4282-9ca1-59613ffb45d8"). InnerVolumeSpecName "kube-api-access-fcddt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.661352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.661265 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc" pod "191ab3e9-ce49-4655-830a-7dc8160e9749" (UID: "191ab3e9-ce49-4655-830a-7dc8160e9749"). InnerVolumeSpecName "kube-api-access-56xqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.661930 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.661912 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9" pod "b0050a22-2ff2-48ac-be21-1fa149bcf98a" (UID: "b0050a22-2ff2-48ac-be21-1fa149bcf98a"). InnerVolumeSpecName "kube-api-access-fzdr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.760769 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.760742 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzdr9\" (UniqueName: \"kubernetes.io/projected/b0050a22-2ff2-48ac-be21-1fa149bcf98a-kube-api-access-fzdr9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.760769 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.760768 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/191ab3e9-ce49-4655-830a-7dc8160e9749-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.760966 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.760777 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcddt\" (UniqueName: \"kubernetes.io/projected/82750f5b-62f3-4282-9ca1-59613ffb45d8-kube-api-access-fcddt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.760966 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.760787 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xqc\" (UniqueName: \"kubernetes.io/projected/191ab3e9-ce49-4655-830a-7dc8160e9749-kube-api-access-56xqc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.760966 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.760795 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82750f5b-62f3-4282-9ca1-59613ffb45d8-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.760966 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:20.760804 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0050a22-2ff2-48ac-be21-1fa149bcf98a-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.076748 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.076714 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw"] Nov 03 21:00:21.077673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.077651 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" Nov 03 21:00:21.077673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.077658 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2" event={"ID":"012a9e62-3609-49b1-8741-0f30c617c8ea","Type":"ContainerDied","Data":"0c9a5827fa42dd436d5d2f0d70bf5d2b821c289a3becc4f8d556e88ffea2b0df"} Nov 03 21:00:21.077879 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.077730 5815 scope.go:122] "RemoveContainer" containerID="043084d4ca037752dd998c2fc85be7983ba8f4ff27ee8d959e9ff2c381932a6c" Nov 03 21:00:21.082236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.082207 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" Nov 03 21:00:21.082236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.082224 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk" event={"ID":"82750f5b-62f3-4282-9ca1-59613ffb45d8","Type":"ContainerDied","Data":"f939e7c1999f1e4839929bfda6cdef3445d44a7e074a48b94b488b123aeda796"} Nov 03 21:00:21.085805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.085785 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" Nov 03 21:00:21.085805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.085791 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg" event={"ID":"191ab3e9-ce49-4655-830a-7dc8160e9749","Type":"ContainerDied","Data":"58b8c4122411e944c77944fe07deb80f22d0c06783a4cd060b36197ab986ea3e"} Nov 03 21:00:21.090851 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.090824 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" event={"ID":"b0050a22-2ff2-48ac-be21-1fa149bcf98a","Type":"ContainerDied","Data":"5b4b6308afc9420cb2a5efc05fe64c3d6cbfcc510b7378ee90ef6fd3b3702cef"} Nov 03 21:00:21.090968 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.090855 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp" Nov 03 21:00:21.092687 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.092633 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" event={"ID":"7da0af6e-dcdc-4120-a6f7-85aea41453c5","Type":"ContainerDied","Data":"e17e3e2ce5a409e8b7703e865744b2480095ccda98efe606b3f9b6cd14e51910"} Nov 03 21:00:21.092881 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.092869 5815 scope.go:122] "RemoveContainer" containerID="e1923eb4afc87cc8c174eb5f14838472e329d055633034af0bb845d226318b8d" Nov 03 21:00:21.093036 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.093000 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x" Nov 03 21:00:21.108633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.108607 5815 generic.go:356] "Generic (PLEG): container finished" podID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerID="defeedfa75bc5d31c0785ea4193ed23a313e207f0a9803e805a9a86cddf34693" exitCode=60 Nov 03 21:00:21.108739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.108652 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn"] Nov 03 21:00:21.108739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.108694 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerDied","Data":"defeedfa75bc5d31c0785ea4193ed23a313e207f0a9803e805a9a86cddf34693"} Nov 03 21:00:21.109121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.109101 5815 scope.go:122] "RemoveContainer" containerID="defeedfa75bc5d31c0785ea4193ed23a313e207f0a9803e805a9a86cddf34693" Nov 03 21:00:21.109467 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:21.109433 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs_odh-ai-gateway-infra(a16e4409-5714-49f4-9af0-c36e0760fcd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" Nov 03 21:00:21.114692 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.114662 5815 scope.go:122] "RemoveContainer" containerID="26ca7239fdd9e010f1c77b5323ac9888d8521f2d02380cb6e3312baf8de9e883" Nov 03 21:00:21.114970 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.114950 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2"] Nov 03 21:00:21.120397 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.120374 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-421604-29895660-jfrx2"] Nov 03 21:00:21.130386 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.130363 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk"] Nov 03 21:00:21.132146 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.132127 5815 scope.go:122] "RemoveContainer" containerID="e46e58ef1c5dc5546cfa75fa704b1147641780439412a9f03c30ac6eba50ecd8" Nov 03 21:00:21.140605 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.140575 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-085566-29895660-qqpsk"] Nov 03 21:00:21.157208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.157149 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x"] Nov 03 21:00:21.169692 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.168872 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr"] Nov 03 21:00:21.171197 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.170465 5815 scope.go:122] "RemoveContainer" containerID="2ee399ce1de21abe777a5d2646deb79cbb1ab951554c323a43895bfe576764fc" Nov 03 21:00:21.176712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.176693 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-7a92d4-29895660-fkh9x"] Nov 03 21:00:21.183297 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.183274 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5"] Nov 03 21:00:21.230442 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.229570 5815 scope.go:122] "RemoveContainer" containerID="de5e1d78926c0f07e63bb99ce892f8fb89a169fa18e9f75a8e179cdf9b33cc58" Nov 03 21:00:21.285672 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.285636 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp"] Nov 03 21:00:21.291722 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.291680 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895660-2j2gp"] Nov 03 21:00:21.322918 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.322899 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:21.345015 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.344955 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg"] Nov 03 21:00:21.350335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.350311 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-37aa95-29895660-8swtg"] Nov 03 21:00:21.374467 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.374435 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca\") pod \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " Nov 03 21:00:21.374674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.374552 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh\" (UniqueName: \"kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh\") pod \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\" (UID: \"c14c22cc-8192-4cc4-86a5-bd11e23de11d\") " Nov 03 21:00:21.376053 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.376022 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca" pod "c14c22cc-8192-4cc4-86a5-bd11e23de11d" (UID: "c14c22cc-8192-4cc4-86a5-bd11e23de11d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.381974 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.381949 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh" pod "c14c22cc-8192-4cc4-86a5-bd11e23de11d" (UID: "c14c22cc-8192-4cc4-86a5-bd11e23de11d"). InnerVolumeSpecName "kube-api-access-q8jjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.460349 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.460323 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:21.464338 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.464312 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:21.470677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.470655 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:21.476572 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.476547 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c22cc-8192-4cc4-86a5-bd11e23de11d-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.476666 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.476574 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jjh\" (UniqueName: \"kubernetes.io/projected/c14c22cc-8192-4cc4-86a5-bd11e23de11d-kube-api-access-q8jjh\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.578172 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.578122 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd\" (UniqueName: \"kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd\") pod \"53905733-a798-40db-8e03-7bfc5e0152a9\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " Nov 03 21:00:21.578411 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.578396 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca\") pod \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " Nov 03 21:00:21.578615 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.578574 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca\") pod \"53905733-a798-40db-8e03-7bfc5e0152a9\" (UID: \"53905733-a798-40db-8e03-7bfc5e0152a9\") " Nov 03 21:00:21.578831 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.578814 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca\") pod \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " Nov 03 21:00:21.579016 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.579002 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75\" (UniqueName: \"kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75\") pod \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\" (UID: \"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e\") " Nov 03 21:00:21.579328 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.579313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4\" (UniqueName: \"kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4\") pod \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\" (UID: \"f1bcb275-c36a-4919-9526-afe5f86e8ad3\") " Nov 03 21:00:21.579524 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.578888 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca" pod "f1bcb275-c36a-4919-9526-afe5f86e8ad3" (UID: "f1bcb275-c36a-4919-9526-afe5f86e8ad3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.579597 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.579472 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca" pod "53905733-a798-40db-8e03-7bfc5e0152a9" (UID: "53905733-a798-40db-8e03-7bfc5e0152a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.579926 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.579902 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca" pod "6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" (UID: "6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.580195 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.580178 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1bcb275-c36a-4919-9526-afe5f86e8ad3-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.580286 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.580273 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53905733-a798-40db-8e03-7bfc5e0152a9-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.580370 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.580360 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.581194 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.581165 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd" pod "53905733-a798-40db-8e03-7bfc5e0152a9" (UID: "53905733-a798-40db-8e03-7bfc5e0152a9"). InnerVolumeSpecName "kube-api-access-f9lcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.582685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.582659 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75" pod "6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" (UID: "6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e"). InnerVolumeSpecName "kube-api-access-bdv75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.582814 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.582790 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4" pod "f1bcb275-c36a-4919-9526-afe5f86e8ad3" (UID: "f1bcb275-c36a-4919-9526-afe5f86e8ad3"). InnerVolumeSpecName "kube-api-access-hqjf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.612812 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.612748 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x"] Nov 03 21:00:21.613372 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613355 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.613372 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613372 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613380 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613385 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613397 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613401 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613411 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613416 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613425 5815 cpu_manager.go:414] "containerMap: removing container" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613430 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613436 5815 cpu_manager.go:414] "containerMap: removing container" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613440 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613456 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613460 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613469 5815 cpu_manager.go:414] "containerMap: removing container" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613473 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613479 5815 cpu_manager.go:414] "containerMap: removing container" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613483 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613523 5815 cpu_manager.go:414] "containerMap: removing container" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613529 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613538 5815 cpu_manager.go:414] "containerMap: removing container" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613542 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.613501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613550 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613554 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613560 5815 cpu_manager.go:414] "containerMap: removing container" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613564 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613577 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613582 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613589 5815 cpu_manager.go:414] "containerMap: removing container" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613593 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613603 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613607 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613616 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613620 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613627 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613631 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613637 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e4001712-c784-4ec4-ba7e-37ba30678285" containerName="revoke-keys" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613642 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4001712-c784-4ec4-ba7e-37ba30678285" containerName="revoke-keys" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613649 5815 cpu_manager.go:414] "containerMap: removing container" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613653 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613660 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613664 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613678 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613682 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613690 5815 cpu_manager.go:414] "containerMap: removing container" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613694 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613700 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613704 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613714 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613718 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613724 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613728 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613735 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613739 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613746 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613751 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613758 5815 cpu_manager.go:414] "containerMap: removing container" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613763 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613770 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613774 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613781 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613785 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613884 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613894 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613900 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613908 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613913 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613918 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613923 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613929 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613933 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613939 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613945 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613950 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613954 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613959 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613964 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2160c8-a5c9-4d9e-a33c-d8add9c2ddc6" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613970 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613974 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613978 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613983 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613988 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613992 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.613998 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614004 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614008 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614012 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614017 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614022 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614027 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614032 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614037 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614045 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614051 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614057 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" containerName="cleanup" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614061 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4001712-c784-4ec4-ba7e-37ba30678285" containerName="revoke-keys" Nov 03 21:00:21.614191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.614066 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:21.618006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.617850 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.622040 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.622017 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:00:21.624989 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.624445 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x"] Nov 03 21:00:21.681607 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.681559 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgw5f\" (UniqueName: \"kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.681941 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.681917 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.682030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.681955 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.682242 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.682216 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdv75\" (UniqueName: \"kubernetes.io/projected/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e-kube-api-access-bdv75\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.682307 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.682249 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjf4\" (UniqueName: \"kubernetes.io/projected/f1bcb275-c36a-4919-9526-afe5f86e8ad3-kube-api-access-hqjf4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.682307 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.682264 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lcd\" (UniqueName: \"kubernetes.io/projected/53905733-a798-40db-8e03-7bfc5e0152a9-kube-api-access-f9lcd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.783792 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.783759 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgw5f\" (UniqueName: \"kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.783939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.783867 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.783939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.783898 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.784679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.784606 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.786426 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.786398 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.796288 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.796266 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgw5f\" (UniqueName: \"kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f\") pod \"maas-api-revoke-sub-be52a41e-q5w9x\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:21.986200 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:21.986168 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:22.124086 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.124052 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" Nov 03 21:00:22.124086 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.124073 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5" event={"ID":"f1bcb275-c36a-4919-9526-afe5f86e8ad3","Type":"ContainerDied","Data":"0ce8b9813738920edf63c555545168d479f234c0f0ec45342320e5c642980a91"} Nov 03 21:00:22.124706 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.124164 5815 scope.go:122] "RemoveContainer" containerID="535923c3bd01dce45fb6fa4c3ed14141a2cca476882f2f637f63921dc7f7b430" Nov 03 21:00:22.136782 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.134781 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" event={"ID":"6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e","Type":"ContainerDied","Data":"2e6ffb9193fde30d6e2c36e90212d59216cc4986f6117fd5a3e2bc7562eaab71"} Nov 03 21:00:22.136782 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.134792 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw" Nov 03 21:00:22.138862 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.138135 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" event={"ID":"c14c22cc-8192-4cc4-86a5-bd11e23de11d","Type":"ContainerDied","Data":"370e7cf10e3846485ccd1f768dc33ccd65a3e880f2d8c93bdd5c4acbf6005684"} Nov 03 21:00:22.138862 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.138249 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn" Nov 03 21:00:22.139498 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.139479 5815 scope.go:122] "RemoveContainer" containerID="440b51521e9ed0a4c0687d52cb74f4140c7934d35cc1d3649c203824a25d2929" Nov 03 21:00:22.143982 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.143962 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" Nov 03 21:00:22.144537 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.143970 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr" event={"ID":"53905733-a798-40db-8e03-7bfc5e0152a9","Type":"ContainerDied","Data":"d5d3b5735118afc45b1bf84ebbc9b1106607ebe54c8660cca90efd9f04ff154e"} Nov 03 21:00:22.153849 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.153821 5815 scope.go:122] "RemoveContainer" containerID="82bab08a409a4c3d71ecbfbd40017f70d0bf6cfe76c945907ae59d680dfe3303" Nov 03 21:00:22.158319 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.158298 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5"] Nov 03 21:00:22.162602 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.162582 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-05473b-29895660-4c2z5"] Nov 03 21:00:22.167051 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.167028 5815 scope.go:122] "RemoveContainer" containerID="72a20649a3b1357fda62ed9ceb5bd2370d2e058f5fc7d6720ff601be81729fe1" Nov 03 21:00:22.182978 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.182953 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw"] Nov 03 21:00:22.186586 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.186565 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895660-q57tw"] Nov 03 21:00:22.203892 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.203870 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn"] Nov 03 21:00:22.209693 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.209664 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14ebbb-29895660-g2gmn"] Nov 03 21:00:22.219718 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.219688 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr"] Nov 03 21:00:22.226619 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.226600 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6350b9-29895660-xbdsr"] Nov 03 21:00:22.272494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.272474 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x"] Nov 03 21:00:22.273836 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:22.273807 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c009781_2002_4e35_b3d6_ea9f3c023008.slice/crio-59aaf1b6515999525d5f30387a748612ae5e90079df6fd17e6fe8b53fe84f5f6 WatchSource:0}: Error finding container 59aaf1b6515999525d5f30387a748612ae5e90079df6fd17e6fe8b53fe84f5f6: Status 404 returned error can't find the container with id 59aaf1b6515999525d5f30387a748612ae5e90079df6fd17e6fe8b53fe84f5f6 Nov 03 21:00:22.338578 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.338551 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs"] Nov 03 21:00:22.515626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.515603 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:22.590081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.590008 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" path="/var/lib/kubelet/pods/012a9e62-3609-49b1-8741-0f30c617c8ea/volumes" Nov 03 21:00:22.590488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.590473 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" path="/var/lib/kubelet/pods/191ab3e9-ce49-4655-830a-7dc8160e9749/volumes" Nov 03 21:00:22.590889 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.590874 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" path="/var/lib/kubelet/pods/53905733-a798-40db-8e03-7bfc5e0152a9/volumes" Nov 03 21:00:22.591596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.591580 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" path="/var/lib/kubelet/pods/6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e/volumes" Nov 03 21:00:22.591963 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.591950 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da0af6e-dcdc-4120-a6f7-85aea41453c5" path="/var/lib/kubelet/pods/7da0af6e-dcdc-4120-a6f7-85aea41453c5/volumes" Nov 03 21:00:22.592334 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.592321 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" path="/var/lib/kubelet/pods/82750f5b-62f3-4282-9ca1-59613ffb45d8/volumes" Nov 03 21:00:22.593007 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.592991 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" path="/var/lib/kubelet/pods/b0050a22-2ff2-48ac-be21-1fa149bcf98a/volumes" Nov 03 21:00:22.593388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.593370 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14c22cc-8192-4cc4-86a5-bd11e23de11d" path="/var/lib/kubelet/pods/c14c22cc-8192-4cc4-86a5-bd11e23de11d/volumes" Nov 03 21:00:22.593815 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.593797 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bcb275-c36a-4919-9526-afe5f86e8ad3" path="/var/lib/kubelet/pods/f1bcb275-c36a-4919-9526-afe5f86e8ad3/volumes" Nov 03 21:00:22.594918 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.594898 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca\") pod \"a16e4409-5714-49f4-9af0-c36e0760fcd3\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " Nov 03 21:00:22.595067 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.595055 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz\" (UniqueName: \"kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz\") pod \"a16e4409-5714-49f4-9af0-c36e0760fcd3\" (UID: \"a16e4409-5714-49f4-9af0-c36e0760fcd3\") " Nov 03 21:00:22.595249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.595227 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca" pod "a16e4409-5714-49f4-9af0-c36e0760fcd3" (UID: "a16e4409-5714-49f4-9af0-c36e0760fcd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:22.595643 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.595622 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16e4409-5714-49f4-9af0-c36e0760fcd3-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:22.665236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.665208 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz" pod "a16e4409-5714-49f4-9af0-c36e0760fcd3" (UID: "a16e4409-5714-49f4-9af0-c36e0760fcd3"). InnerVolumeSpecName "kube-api-access-jhqjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:22.696995 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:22.696814 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqjz\" (UniqueName: \"kubernetes.io/projected/a16e4409-5714-49f4-9af0-c36e0760fcd3-kube-api-access-jhqjz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.158581 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.158548 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" event={"ID":"5c009781-2002-4e35-b3d6-ea9f3c023008","Type":"ContainerStarted","Data":"5c0c8c1bcdb623f1798937d9a08df366e3d917e3e8bfc182ce3f34b5748b412f"} Nov 03 21:00:23.159015 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.158600 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" event={"ID":"5c009781-2002-4e35-b3d6-ea9f3c023008","Type":"ContainerStarted","Data":"59aaf1b6515999525d5f30387a748612ae5e90079df6fd17e6fe8b53fe84f5f6"} Nov 03 21:00:23.161091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.161071 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" Nov 03 21:00:23.161233 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.161116 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs" event={"ID":"a16e4409-5714-49f4-9af0-c36e0760fcd3","Type":"ContainerDied","Data":"6b3569578ad302eb65c27634957f6b3b8527c781763f7d5fb091a0fdaf1cac39"} Nov 03 21:00:23.161233 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.161172 5815 scope.go:122] "RemoveContainer" containerID="defeedfa75bc5d31c0785ea4193ed23a313e207f0a9803e805a9a86cddf34693" Nov 03 21:00:23.178859 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.177933 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" podStartSLOduration=2.17791871 podStartE2EDuration="2.17791871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:23.177303506 +0000 UTC m=+3532.707649179" watchObservedRunningTime="2026-11-03 21:00:23.17791871 +0000 UTC m=+3532.708264385" Nov 03 21:00:23.195905 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.195882 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs"] Nov 03 21:00:23.201078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:23.201057 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-dce009-29895660-6h4qs"] Nov 03 21:00:24.175192 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:24.175157 5815 generic.go:356] "Generic (PLEG): container finished" podID="5c009781-2002-4e35-b3d6-ea9f3c023008" containerID="5c0c8c1bcdb623f1798937d9a08df366e3d917e3e8bfc182ce3f34b5748b412f" exitCode=0 Nov 03 21:00:24.175631 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:24.175248 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" event={"ID":"5c009781-2002-4e35-b3d6-ea9f3c023008","Type":"ContainerDied","Data":"5c0c8c1bcdb623f1798937d9a08df366e3d917e3e8bfc182ce3f34b5748b412f"} Nov 03 21:00:24.590167 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:24.590127 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" path="/var/lib/kubelet/pods/a16e4409-5714-49f4-9af0-c36e0760fcd3/volumes" Nov 03 21:00:25.666143 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.666113 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:25.821241 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.821149 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca\") pod \"5c009781-2002-4e35-b3d6-ea9f3c023008\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " Nov 03 21:00:25.821241 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.821240 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f\" (UniqueName: \"kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f\") pod \"5c009781-2002-4e35-b3d6-ea9f3c023008\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " Nov 03 21:00:25.821475 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.821422 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca\") pod \"5c009781-2002-4e35-b3d6-ea9f3c023008\" (UID: \"5c009781-2002-4e35-b3d6-ea9f3c023008\") " Nov 03 21:00:25.821662 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.821637 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca" pod "5c009781-2002-4e35-b3d6-ea9f3c023008" (UID: "5c009781-2002-4e35-b3d6-ea9f3c023008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:25.821996 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.821978 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.824093 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.824063 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f" pod "5c009781-2002-4e35-b3d6-ea9f3c023008" (UID: "5c009781-2002-4e35-b3d6-ea9f3c023008"). InnerVolumeSpecName "kube-api-access-mgw5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:25.824233 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.824178 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca" pod "5c009781-2002-4e35-b3d6-ea9f3c023008" (UID: "5c009781-2002-4e35-b3d6-ea9f3c023008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:25.932583 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.932546 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c009781-2002-4e35-b3d6-ea9f3c023008-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.932860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:25.932826 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgw5f\" (UniqueName: \"kubernetes.io/projected/5c009781-2002-4e35-b3d6-ea9f3c023008-kube-api-access-mgw5f\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:26.261962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:26.261931 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" Nov 03 21:00:26.262180 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:26.262147 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x" event={"ID":"5c009781-2002-4e35-b3d6-ea9f3c023008","Type":"ContainerDied","Data":"59aaf1b6515999525d5f30387a748612ae5e90079df6fd17e6fe8b53fe84f5f6"} Nov 03 21:00:26.262283 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:26.262227 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59aaf1b6515999525d5f30387a748612ae5e90079df6fd17e6fe8b53fe84f5f6" Nov 03 21:00:34.002746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:33.998398 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb"] Nov 03 21:00:38.006928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:38.004884 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7"] Nov 03 21:00:38.006928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:38.005587 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" containerID="cri-o://e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594" gracePeriod=30 Nov 03 21:00:40.565784 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:40.565747 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj"] Nov 03 21:00:43.308149 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.308119 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 21:00:43.357875 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.357837 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358070 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.357903 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358070 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.357972 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358070 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.358032 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358070 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.358061 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358406 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.358160 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358406 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.358219 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358406 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.358300 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.358406 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.358397 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data\") pod \"e6c35e48-442e-4a6b-8796-3d086c92b955\" (UID: \"e6c35e48-442e-4a6b-8796-3d086c92b955\") " Nov 03 21:00:43.359560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.359244 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.359742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.359709 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.359943 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.359918 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.360073 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.360046 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:43.360341 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.360317 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.362242 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.362217 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:00:43.362998 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.362239 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.362998 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.362271 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:43.362998 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.362737 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc" pod "e6c35e48-442e-4a6b-8796-3d086c92b955" (UID: "e6c35e48-442e-4a6b-8796-3d086c92b955"). InnerVolumeSpecName "kube-api-access-sm8vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:43.456533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.453775 5815 generic.go:356] "Generic (PLEG): container finished" podID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerID="e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594" exitCode=0 Nov 03 21:00:43.456533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.453904 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" event={"ID":"e6c35e48-442e-4a6b-8796-3d086c92b955","Type":"ContainerDied","Data":"e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594"} Nov 03 21:00:43.456533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.453962 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" event={"ID":"e6c35e48-442e-4a6b-8796-3d086c92b955","Type":"ContainerDied","Data":"62a5538a62168459eebe340b25e59cee5ce147d8ba32f25ca2b0de9456b96fae"} Nov 03 21:00:43.456533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.453991 5815 scope.go:122] "RemoveContainer" containerID="e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594" Nov 03 21:00:43.456533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.454156 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460340 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460383 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460399 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460420 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460439 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460455 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460471 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6c35e48-442e-4a6b-8796-3d086c92b955-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460492 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm8vc\" (UniqueName: \"kubernetes.io/projected/e6c35e48-442e-4a6b-8796-3d086c92b955-kube-api-access-sm8vc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.460555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.460526 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6c35e48-442e-4a6b-8796-3d086c92b955-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.485173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.484390 5815 scope.go:122] "RemoveContainer" containerID="e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594" Nov 03 21:00:43.485173 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:43.484828 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594\": container with ID starting with e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594 not found: ID does not exist" containerID="e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594" Nov 03 21:00:43.485173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.484869 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594"} err="failed to get container status \"e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594\": rpc error: code = NotFound desc = could not find container \"e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594\": container with ID starting with e6d9ad1d789d9cd6a0c9e891cd632095d098baab0b6c039ef0891d97ae545594 not found: ID does not exist" Nov 03 21:00:43.506739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.506711 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7"] Nov 03 21:00:43.511689 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:43.511670 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-lszn7"] Nov 03 21:00:43.854087 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:43.854053 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:00:44.589728 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:44.589687 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" path="/var/lib/kubelet/pods/e6c35e48-442e-4a6b-8796-3d086c92b955/volumes" Nov 03 21:00:46.686149 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:46.686113 5815 scope.go:122] "RemoveContainer" containerID="c73b9659726d44dd5b70826bbad8f94916b3a13b06137e8d99b59347bad351c1" Nov 03 21:00:46.699109 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:46.699086 5815 scope.go:122] "RemoveContainer" containerID="57477604025da5de2e7072e080ae6b317b9ec41c755917ab685fbdeb1979e2d5" Nov 03 21:00:46.924167 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:46.924131 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn"] Nov 03 21:00:49.032073 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.032034 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" containerID="cri-o://30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2" gracePeriod=45 Nov 03 21:00:49.231546 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.228992 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.154:8000/ready\": dial tcp 10.132.1.154:8000: connect: connection refused" Nov 03 21:00:49.379677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.379650 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 21:00:49.490911 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.490873 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home\") pod \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " Nov 03 21:00:49.491121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.490950 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs\") pod \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " Nov 03 21:00:49.491121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491055 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir\") pod \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " Nov 03 21:00:49.491245 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491119 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5\" (UniqueName: \"kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5\") pod \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " Nov 03 21:00:49.491245 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491154 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home" pod "6772c2ad-ac79-45ea-bf6a-e63c817da3e8" (UID: "6772c2ad-ac79-45ea-bf6a-e63c817da3e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:49.491355 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491325 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir" pod "6772c2ad-ac79-45ea-bf6a-e63c817da3e8" (UID: "6772c2ad-ac79-45ea-bf6a-e63c817da3e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:49.491471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491384 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm\") pod \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " Nov 03 21:00:49.491471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491415 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache\") pod \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\" (UID: \"6772c2ad-ac79-45ea-bf6a-e63c817da3e8\") " Nov 03 21:00:49.491775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.491733 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache" pod "6772c2ad-ac79-45ea-bf6a-e63c817da3e8" (UID: "6772c2ad-ac79-45ea-bf6a-e63c817da3e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:49.492326 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.492300 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.492420 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.492333 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.492420 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.492351 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.493641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.493605 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs" pod "6772c2ad-ac79-45ea-bf6a-e63c817da3e8" (UID: "6772c2ad-ac79-45ea-bf6a-e63c817da3e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:49.494095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.494072 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm" pod "6772c2ad-ac79-45ea-bf6a-e63c817da3e8" (UID: "6772c2ad-ac79-45ea-bf6a-e63c817da3e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:49.494206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.494184 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5" pod "6772c2ad-ac79-45ea-bf6a-e63c817da3e8" (UID: "6772c2ad-ac79-45ea-bf6a-e63c817da3e8"). InnerVolumeSpecName "kube-api-access-cpxn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:49.545071 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.545048 5815 generic.go:356] "Generic (PLEG): container finished" podID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerID="30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2" exitCode=0 Nov 03 21:00:49.545179 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.545096 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" Nov 03 21:00:49.545179 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.545098 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" event={"ID":"6772c2ad-ac79-45ea-bf6a-e63c817da3e8","Type":"ContainerDied","Data":"30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2"} Nov 03 21:00:49.545274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.545198 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb" event={"ID":"6772c2ad-ac79-45ea-bf6a-e63c817da3e8","Type":"ContainerDied","Data":"b4110352f533e34f7cc98df99cc1f926e877ce0ec3cfac48d61f95c7a3fec52a"} Nov 03 21:00:49.545274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.545219 5815 scope.go:122] "RemoveContainer" containerID="30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2" Nov 03 21:00:49.557276 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.557257 5815 scope.go:122] "RemoveContainer" containerID="30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2" Nov 03 21:00:49.557596 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:49.557573 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2\": container with ID starting with 30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2 not found: ID does not exist" containerID="30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2" Nov 03 21:00:49.557678 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.557602 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2"} err="failed to get container status \"30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2\": rpc error: code = NotFound desc = could not find container \"30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2\": container with ID starting with 30993901349cfc18fca013673135e937c4d465d43417ae6005d85703988d57e2 not found: ID does not exist" Nov 03 21:00:49.570891 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.570870 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb"] Nov 03 21:00:49.576446 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.576427 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/rate-test-model-ba89e0-kserve-dffdb7669-6bfkb"] Nov 03 21:00:49.594056 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.594030 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxn5\" (UniqueName: \"kubernetes.io/projected/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-kube-api-access-cpxn5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.594153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.594070 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.594153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.594086 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6772c2ad-ac79-45ea-bf6a-e63c817da3e8-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.791088 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.791047 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns"] Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792114 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792139 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792160 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792170 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792189 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792197 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792218 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792226 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911fa7a-c5c5-4650-96cb-d2d4ce5d96c5" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792237 5815 cpu_manager.go:414] "containerMap: removing container" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792244 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792261 5815 cpu_manager.go:414] "containerMap: removing container" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792269 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792282 5815 cpu_manager.go:414] "containerMap: removing container" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792289 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ab3e9-ce49-4655-830a-7dc8160e9749" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792300 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792307 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0050a22-2ff2-48ac-be21-1fa149bcf98a" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792317 5815 cpu_manager.go:414] "containerMap: removing container" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792325 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012a9e62-3609-49b1-8741-0f30c617c8ea" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792352 5815 cpu_manager.go:414] "containerMap: removing container" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792360 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792375 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5c009781-2002-4e35-b3d6-ea9f3c023008" containerName="revoke-keys" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792382 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c009781-2002-4e35-b3d6-ea9f3c023008" containerName="revoke-keys" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792395 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792403 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792415 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792422 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792437 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.792445 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793128 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793148 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c009781-2002-4e35-b3d6-ea9f3c023008" containerName="revoke-keys" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793162 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793180 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="93056b14-35ff-4299-ae03-ce994d5f31f3" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793194 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" containerName="main" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793205 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="82750f5b-62f3-4282-9ca1-59613ffb45d8" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793215 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1863d0ed-1093-4ba8-adbe-bb0ce554ff8e" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793226 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d8fd06c-af59-4b1f-b8b9-8f5e26a8422e" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793236 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="53905733-a798-40db-8e03-7bfc5e0152a9" containerName="cleanup" Nov 03 21:00:49.799596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.793247 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c35e48-442e-4a6b-8796-3d086c92b955" containerName="istio-proxy" Nov 03 21:00:49.822498 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.817826 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.823381 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.823330 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"kube-root-ca.crt\"" Nov 03 21:00:49.823573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.823490 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"sub-test-model-ba89e0-kserve-self-signed-certs\"" Nov 03 21:00:49.823684 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.823667 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"default-dockercfg-pr7n2\"" Nov 03 21:00:49.832571 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.832542 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns"] Nov 03 21:00:49.911260 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.911225 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.911433 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.911405 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.911535 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.911457 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.911638 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.911619 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.911730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.911713 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.911877 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.911862 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:49.923370 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:49.923344 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ba89e0\"/\"openshift-service-ca.crt\"" Nov 03 21:00:50.016555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.014543 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.016555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.014636 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.016555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.014812 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.016555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.014883 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.016555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.014953 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.016555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.014981 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.028547 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.024208 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.028547 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.025015 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.028547 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.025269 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.070414 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.070331 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.071216 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.071195 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.080825 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.080795 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h\") pod \"sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.153201 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.153164 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:50.460775 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:00:50.460742 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72f29c2b_2da7_45b6_bace_92b83127a214.slice/crio-a162b571225996550fc9628522f9a229223a66072b26c1678b5eb494d00be17d WatchSource:0}: Error finding container a162b571225996550fc9628522f9a229223a66072b26c1678b5eb494d00be17d: Status 404 returned error can't find the container with id a162b571225996550fc9628522f9a229223a66072b26c1678b5eb494d00be17d Nov 03 21:00:50.462430 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.462401 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns"] Nov 03 21:00:50.597007 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.596965 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6772c2ad-ac79-45ea-bf6a-e63c817da3e8" path="/var/lib/kubelet/pods/6772c2ad-ac79-45ea-bf6a-e63c817da3e8/volumes" Nov 03 21:00:50.597429 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:50.597406 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" event={"ID":"72f29c2b-2da7-45b6-bace-92b83127a214","Type":"ContainerStarted","Data":"a162b571225996550fc9628522f9a229223a66072b26c1678b5eb494d00be17d"} Nov 03 21:00:51.611728 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:51.611672 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" event={"ID":"72f29c2b-2da7-45b6-bace-92b83127a214","Type":"ContainerStarted","Data":"684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828"} Nov 03 21:00:51.635110 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:51.635055 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" podStartSLOduration=2.635041848 podStartE2EDuration="2.635041848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:51.631355436 +0000 UTC m=+3561.161701107" watchObservedRunningTime="2026-11-03 21:00:51.635041848 +0000 UTC m=+3561.165387519" Nov 03 21:00:51.924620 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:51.924471 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerName="payload-pre-processing" containerID="cri-o://14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d" gracePeriod=25 Nov 03 21:00:52.165344 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:52.160444 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:00:55.600879 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:55.600816 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" podUID="b0399410-b41e-446e-8300-22e4486211f7" containerName="main" containerID="cri-o://54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71" gracePeriod=45 Nov 03 21:00:55.939366 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:55.939327 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 21:00:56.057531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.053376 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir\") pod \"b0399410-b41e-446e-8300-22e4486211f7\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " Nov 03 21:00:56.057531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.053484 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r\" (UniqueName: \"kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r\") pod \"b0399410-b41e-446e-8300-22e4486211f7\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " Nov 03 21:00:56.057531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.053612 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs\") pod \"b0399410-b41e-446e-8300-22e4486211f7\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " Nov 03 21:00:56.057531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.053695 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache\") pod \"b0399410-b41e-446e-8300-22e4486211f7\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " Nov 03 21:00:56.057531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.053738 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home\") pod \"b0399410-b41e-446e-8300-22e4486211f7\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " Nov 03 21:00:56.057531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.053812 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm\") pod \"b0399410-b41e-446e-8300-22e4486211f7\" (UID: \"b0399410-b41e-446e-8300-22e4486211f7\") " Nov 03 21:00:56.091531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.087747 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache" pod "b0399410-b41e-446e-8300-22e4486211f7" (UID: "b0399410-b41e-446e-8300-22e4486211f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:56.091531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.087911 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home" pod "b0399410-b41e-446e-8300-22e4486211f7" (UID: "b0399410-b41e-446e-8300-22e4486211f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:56.091531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.088042 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir" pod "b0399410-b41e-446e-8300-22e4486211f7" (UID: "b0399410-b41e-446e-8300-22e4486211f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:56.125288 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.125178 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm" pod "b0399410-b41e-446e-8300-22e4486211f7" (UID: "b0399410-b41e-446e-8300-22e4486211f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:56.128068 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.128037 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r" pod "b0399410-b41e-446e-8300-22e4486211f7" (UID: "b0399410-b41e-446e-8300-22e4486211f7"). InnerVolumeSpecName "kube-api-access-spn2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:56.141935 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.141903 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs" pod "b0399410-b41e-446e-8300-22e4486211f7" (UID: "b0399410-b41e-446e-8300-22e4486211f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:56.155385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.155304 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:56.155640 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.155622 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:56.155760 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.155748 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:56.155947 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.155934 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0399410-b41e-446e-8300-22e4486211f7-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:56.156056 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.156044 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spn2r\" (UniqueName: \"kubernetes.io/projected/b0399410-b41e-446e-8300-22e4486211f7-kube-api-access-spn2r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:56.156142 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.156131 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0399410-b41e-446e-8300-22e4486211f7-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:56.696123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.696092 5815 generic.go:356] "Generic (PLEG): container finished" podID="b0399410-b41e-446e-8300-22e4486211f7" containerID="54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71" exitCode=0 Nov 03 21:00:56.696561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.696138 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" Nov 03 21:00:56.696561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.696148 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" event={"ID":"b0399410-b41e-446e-8300-22e4486211f7","Type":"ContainerDied","Data":"54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71"} Nov 03 21:00:56.696561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.696178 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj" event={"ID":"b0399410-b41e-446e-8300-22e4486211f7","Type":"ContainerDied","Data":"65e6c3cb5017289c9dd90d550dae6010f6788fb1c2c3cea9ae8dbd2c8c51c995"} Nov 03 21:00:56.696561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.696193 5815 scope.go:122] "RemoveContainer" containerID="54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71" Nov 03 21:00:56.707333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.707313 5815 scope.go:122] "RemoveContainer" containerID="54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71" Nov 03 21:00:56.707595 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:00:56.707569 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71\": container with ID starting with 54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71 not found: ID does not exist" containerID="54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71" Nov 03 21:00:56.707714 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.707602 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71"} err="failed to get container status \"54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71\": rpc error: code = NotFound desc = could not find container \"54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71\": container with ID starting with 54da047d0d7b5a3b7ee14c32c03a236b5f9fa3e6693615b140c5ef55a92abd71 not found: ID does not exist" Nov 03 21:00:56.720122 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.720097 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj"] Nov 03 21:00:56.724267 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:56.724246 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/rate-test-model-99a8fe-kserve-64cff5f97c-d69lj"] Nov 03 21:00:58.588457 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:58.588424 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0399410-b41e-446e-8300-22e4486211f7" path="/var/lib/kubelet/pods/b0399410-b41e-446e-8300-22e4486211f7/volumes" Nov 03 21:00:58.744187 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:00:58.744154 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:01:00.153722 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:00.153678 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:01:00.167169 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:00.167138 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:01:00.781150 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:00.781118 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:01:04.195753 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:04.195714 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t"] Nov 03 21:01:04.214991 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:04.209413 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh"] Nov 03 21:01:08.744346 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:08.744312 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:01:09.196929 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.196826 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" podUID="3955adb0-0d7a-4707-90c5-194b5b0e9793" containerName="payload-processing" containerID="cri-o://9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4" gracePeriod=25 Nov 03 21:01:09.210454 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.210420 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerName="payload-pre-processing" containerID="cri-o://81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07" gracePeriod=25 Nov 03 21:01:09.539113 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.539088 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 21:01:09.678730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.678692 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc\" (UniqueName: \"kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc\") pod \"3955adb0-0d7a-4707-90c5-194b5b0e9793\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " Nov 03 21:01:09.678989 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.678972 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume\") pod \"3955adb0-0d7a-4707-90c5-194b5b0e9793\" (UID: \"3955adb0-0d7a-4707-90c5-194b5b0e9793\") " Nov 03 21:01:09.679586 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.679557 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume" pod "3955adb0-0d7a-4707-90c5-194b5b0e9793" (UID: "3955adb0-0d7a-4707-90c5-194b5b0e9793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:09.681216 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.681190 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc" pod "3955adb0-0d7a-4707-90c5-194b5b0e9793" (UID: "3955adb0-0d7a-4707-90c5-194b5b0e9793"). InnerVolumeSpecName "kube-api-access-smcpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:09.781398 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.781329 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3955adb0-0d7a-4707-90c5-194b5b0e9793-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:09.781398 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.781353 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smcpc\" (UniqueName: \"kubernetes.io/projected/3955adb0-0d7a-4707-90c5-194b5b0e9793-kube-api-access-smcpc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:09.866053 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.866020 5815 generic.go:356] "Generic (PLEG): container finished" podID="3955adb0-0d7a-4707-90c5-194b5b0e9793" containerID="9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4" exitCode=0 Nov 03 21:01:09.866238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.866059 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" Nov 03 21:01:09.866238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.866111 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" event={"ID":"3955adb0-0d7a-4707-90c5-194b5b0e9793","Type":"ContainerDied","Data":"9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4"} Nov 03 21:01:09.866238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.866182 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t" event={"ID":"3955adb0-0d7a-4707-90c5-194b5b0e9793","Type":"ContainerDied","Data":"3347594651d37401dab2542e58c4f4a9123989940dbcd59c0b955c9b4378a534"} Nov 03 21:01:09.866238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.866212 5815 scope.go:122] "RemoveContainer" containerID="9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4" Nov 03 21:01:09.879177 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.879156 5815 scope.go:122] "RemoveContainer" containerID="9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4" Nov 03 21:01:09.879470 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:09.879451 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4\": container with ID starting with 9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4 not found: ID does not exist" containerID="9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4" Nov 03 21:01:09.879615 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.879477 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4"} err="failed to get container status \"9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4\": rpc error: code = NotFound desc = could not find container \"9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4\": container with ID starting with 9762c09fae227fb5f749d74658949bf1632312fffee4356096005d0cac55ade4 not found: ID does not exist" Nov 03 21:01:09.901583 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.901551 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t"] Nov 03 21:01:09.901776 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:09.901612 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-05473b-7ff86fff6-wtw2t"] Nov 03 21:01:10.069092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:10.069014 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:01:10.607041 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:10.607011 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3955adb0-0d7a-4707-90c5-194b5b0e9793" path="/var/lib/kubelet/pods/3955adb0-0d7a-4707-90c5-194b5b0e9793/volumes" Nov 03 21:01:11.262692 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:11.262655 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:01:12.034594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.031552 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb"] Nov 03 21:01:12.038376 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.038347 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5dba8c-fnmcb"] Nov 03 21:01:12.365896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.365863 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 21:01:12.442926 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.442896 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p\" (UniqueName: \"kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p\") pod \"ca54681f-3a06-45ef-ab20-5d0607c178dc\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " Nov 03 21:01:12.443096 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.442992 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume\") pod \"ca54681f-3a06-45ef-ab20-5d0607c178dc\" (UID: \"ca54681f-3a06-45ef-ab20-5d0607c178dc\") " Nov 03 21:01:12.443420 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.443393 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume" pod "ca54681f-3a06-45ef-ab20-5d0607c178dc" (UID: "ca54681f-3a06-45ef-ab20-5d0607c178dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:12.443496 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.443478 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54681f-3a06-45ef-ab20-5d0607c178dc-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:12.445240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.445217 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p" pod "ca54681f-3a06-45ef-ab20-5d0607c178dc" (UID: "ca54681f-3a06-45ef-ab20-5d0607c178dc"). InnerVolumeSpecName "kube-api-access-dhj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:12.544297 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.544232 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhj6p\" (UniqueName: \"kubernetes.io/projected/ca54681f-3a06-45ef-ab20-5d0607c178dc-kube-api-access-dhj6p\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:12.587842 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.587807 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7067bb27-b8ed-4fe7-b4eb-fc1d2f804155" path="/var/lib/kubelet/pods/7067bb27-b8ed-4fe7-b4eb-fc1d2f804155/volumes" Nov 03 21:01:12.959013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.958926 5815 generic.go:356] "Generic (PLEG): container finished" podID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerID="14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d" exitCode=0 Nov 03 21:01:12.959268 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.958975 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" Nov 03 21:01:12.959520 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.958993 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" event={"ID":"ca54681f-3a06-45ef-ab20-5d0607c178dc","Type":"ContainerDied","Data":"14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d"} Nov 03 21:01:12.959682 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.959665 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn" event={"ID":"ca54681f-3a06-45ef-ab20-5d0607c178dc","Type":"ContainerDied","Data":"c224669e213cafc4f40f82c2ccab797a5a522bc9cdcb7bf4212f3bd1212d91e6"} Nov 03 21:01:12.959811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.959796 5815 scope.go:122] "RemoveContainer" containerID="14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d" Nov 03 21:01:12.974304 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.974283 5815 scope.go:122] "RemoveContainer" containerID="14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d" Nov 03 21:01:12.974706 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:12.974683 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d\": container with ID starting with 14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d not found: ID does not exist" containerID="14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d" Nov 03 21:01:12.974827 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.974718 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d"} err="failed to get container status \"14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d\": rpc error: code = NotFound desc = could not find container \"14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d\": container with ID starting with 14e9c360fe4e844eb44bdb65f34f7802959726e921707e469a2cd2da9508a08d not found: ID does not exist" Nov 03 21:01:12.989633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.989596 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn"] Nov 03 21:01:12.994203 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:12.994168 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-dce009-748986cfc5-hbbvn"] Nov 03 21:01:13.858124 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:13.858091 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:01:14.618531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:14.618004 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" path="/var/lib/kubelet/pods/ca54681f-3a06-45ef-ab20-5d0607c178dc/volumes" Nov 03 21:01:15.405984 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.405948 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-5618ac"] Nov 03 21:01:15.406915 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.406895 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3955adb0-0d7a-4707-90c5-194b5b0e9793" containerName="payload-processing" Nov 03 21:01:15.406995 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.406918 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3955adb0-0d7a-4707-90c5-194b5b0e9793" containerName="payload-processing" Nov 03 21:01:15.406995 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.406946 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b0399410-b41e-446e-8300-22e4486211f7" containerName="main" Nov 03 21:01:15.406995 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.406954 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0399410-b41e-446e-8300-22e4486211f7" containerName="main" Nov 03 21:01:15.407091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407015 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:01:15.407091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407022 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:01:15.407091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407033 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerName="payload-pre-processing" Nov 03 21:01:15.407091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407041 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerName="payload-pre-processing" Nov 03 21:01:15.407222 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407193 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16e4409-5714-49f4-9af0-c36e0760fcd3" containerName="cleanup" Nov 03 21:01:15.407222 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407216 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca54681f-3a06-45ef-ab20-5d0607c178dc" containerName="payload-pre-processing" Nov 03 21:01:15.407302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407231 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3955adb0-0d7a-4707-90c5-194b5b0e9793" containerName="payload-processing" Nov 03 21:01:15.407302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.407240 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0399410-b41e-446e-8300-22e4486211f7" containerName="main" Nov 03 21:01:15.412891 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.412871 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:15.413026 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.412977 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-5618ac"] Nov 03 21:01:15.417050 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.417026 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6p5pq\"" Nov 03 21:01:15.421413 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.421386 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dgl\" (UniqueName: \"kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl\") pod \"test-curl-7573-5618ac\" (UID: \"e2a31c27-ece2-4af3-865d-20f1f51c4583\") " pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:15.523577 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.523538 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dgl\" (UniqueName: \"kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl\") pod \"test-curl-7573-5618ac\" (UID: \"e2a31c27-ece2-4af3-865d-20f1f51c4583\") " pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:15.536000 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.535970 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dgl\" (UniqueName: \"kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl\") pod \"test-curl-7573-5618ac\" (UID: \"e2a31c27-ece2-4af3-865d-20f1f51c4583\") " pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:15.728611 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:15.728582 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:16.014434 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:01:16.014405 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2a31c27_ece2_4af3_865d_20f1f51c4583.slice/crio-fd1cbb687898bae242ecdeccbbda75f713399eca7359c8253220424dc75a27fa WatchSource:0}: Error finding container fd1cbb687898bae242ecdeccbbda75f713399eca7359c8253220424dc75a27fa: Status 404 returned error can't find the container with id fd1cbb687898bae242ecdeccbbda75f713399eca7359c8253220424dc75a27fa Nov 03 21:01:16.015798 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:16.015759 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-5618ac"] Nov 03 21:01:17.004088 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.004052 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-5618ac" event={"ID":"e2a31c27-ece2-4af3-865d-20f1f51c4583","Type":"ContainerStarted","Data":"7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793"} Nov 03 21:01:17.004495 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.004113 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-5618ac" event={"ID":"e2a31c27-ece2-4af3-865d-20f1f51c4583","Type":"ContainerStarted","Data":"fd1cbb687898bae242ecdeccbbda75f713399eca7359c8253220424dc75a27fa"} Nov 03 21:01:17.038531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.028074 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7573-5618ac" podStartSLOduration=2.028055484 podStartE2EDuration="2.028055484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:17.027932852 +0000 UTC m=+3586.558278526" watchObservedRunningTime="2026-11-03 21:01:17.028055484 +0000 UTC m=+3586.558401157" Nov 03 21:01:17.691271 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.691232 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-5618ac"] Nov 03 21:01:17.699215 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.699183 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-5618ac"] Nov 03 21:01:17.843866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.843833 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7573-d81fc0"] Nov 03 21:01:17.844739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.844720 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e2a31c27-ece2-4af3-865d-20f1f51c4583" containerName="test-curl-7573-5618ac" Nov 03 21:01:17.844796 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.844742 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a31c27-ece2-4af3-865d-20f1f51c4583" containerName="test-curl-7573-5618ac" Nov 03 21:01:17.844929 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.844918 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a31c27-ece2-4af3-865d-20f1f51c4583" containerName="test-curl-7573-5618ac" Nov 03 21:01:17.849217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.849196 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:17.853442 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.852630 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-d81fc0"] Nov 03 21:01:17.959333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:17.959253 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttmp\" (UniqueName: \"kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp\") pod \"test-curl-7573-d81fc0\" (UID: \"20358686-575e-4f17-a545-7e12558b1559\") " pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:18.060875 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:18.060844 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttmp\" (UniqueName: \"kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp\") pod \"test-curl-7573-d81fc0\" (UID: \"20358686-575e-4f17-a545-7e12558b1559\") " pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:18.073351 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:18.073320 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttmp\" (UniqueName: \"kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp\") pod \"test-curl-7573-d81fc0\" (UID: \"20358686-575e-4f17-a545-7e12558b1559\") " pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:18.168570 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:18.168473 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl"] Nov 03 21:01:18.168765 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:18.168746 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" containerID="cri-o://62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951" gracePeriod=30 Nov 03 21:01:18.170475 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:18.170455 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:18.510262 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:18.508123 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7573-d81fc0"] Nov 03 21:01:19.037123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:19.037081 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-d81fc0" event={"ID":"20358686-575e-4f17-a545-7e12558b1559","Type":"ContainerStarted","Data":"aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4"} Nov 03 21:01:19.037333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:19.037137 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7573-d81fc0" event={"ID":"20358686-575e-4f17-a545-7e12558b1559","Type":"ContainerStarted","Data":"4746e3358554733408f5844e102204c07604bd38f8707577991c5a9a15eba16c"} Nov 03 21:01:19.037333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:19.037312 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7573-5618ac" podUID="e2a31c27-ece2-4af3-865d-20f1f51c4583" containerName="test-curl-7573-5618ac" containerID="cri-o://7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793" gracePeriod=2 Nov 03 21:01:19.055018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:19.054976 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7573-d81fc0" podStartSLOduration=2.054962981 podStartE2EDuration="2.054962981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:19.053847394 +0000 UTC m=+3588.584193068" watchObservedRunningTime="2026-11-03 21:01:19.054962981 +0000 UTC m=+3588.585308652" Nov 03 21:01:19.736541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:19.736480 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7573-d81fc0"] Nov 03 21:01:19.742258 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:19.742225 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7573-d81fc0"] Nov 03 21:01:20.069048 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.068957 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:01:20.248849 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.248818 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k"] Nov 03 21:01:20.249455 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.249434 5815 cpu_manager.go:414] "containerMap: removing container" podUID="20358686-575e-4f17-a545-7e12558b1559" containerName="test-curl-7573-d81fc0" Nov 03 21:01:20.249537 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.249456 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20358686-575e-4f17-a545-7e12558b1559" containerName="test-curl-7573-d81fc0" Nov 03 21:01:20.249606 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.249595 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="20358686-575e-4f17-a545-7e12558b1559" containerName="test-curl-7573-d81fc0" Nov 03 21:01:20.253130 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.253111 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.257496 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.257461 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:20.260802 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.260772 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:01:20.273964 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.273322 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k"] Nov 03 21:01:20.407010 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.406926 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.407186 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.407110 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.407310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.407291 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675bk\" (UniqueName: \"kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.508334 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.508295 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.508537 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.508439 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675bk\" (UniqueName: \"kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.508599 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.508554 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.509320 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.509288 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.510960 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.510935 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.520846 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.520824 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-675bk\" (UniqueName: \"kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk\") pod \"maas-api-revoke-sub-01b19140-tlj5k\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.619712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.619683 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:20.908001 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:20.907971 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k"] Nov 03 21:01:20.909349 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:01:20.909322 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ab6af4_cfb9_4054_a616_174cc898d9b0.slice/crio-0cf1136e1eedacd9e8ac1d52ead1b0c234e43483c97e9bf713162d445e473e6d WatchSource:0}: Error finding container 0cf1136e1eedacd9e8ac1d52ead1b0c234e43483c97e9bf713162d445e473e6d: Status 404 returned error can't find the container with id 0cf1136e1eedacd9e8ac1d52ead1b0c234e43483c97e9bf713162d445e473e6d Nov 03 21:01:21.057933 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:21.057907 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" event={"ID":"a0ab6af4-cfb9-4054-a616-174cc898d9b0","Type":"ContainerStarted","Data":"0cf1136e1eedacd9e8ac1d52ead1b0c234e43483c97e9bf713162d445e473e6d"} Nov 03 21:01:21.058232 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:21.058206 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7573-d81fc0" podUID="20358686-575e-4f17-a545-7e12558b1559" containerName="test-curl-7573-d81fc0" containerID="cri-o://aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4" gracePeriod=2 Nov 03 21:01:21.240738 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:21.240714 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:21.319341 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:21.319313 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl\" (UniqueName: \"kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl\") pod \"e2a31c27-ece2-4af3-865d-20f1f51c4583\" (UID: \"e2a31c27-ece2-4af3-865d-20f1f51c4583\") " Nov 03 21:01:21.321636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:21.321609 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl" pod "e2a31c27-ece2-4af3-865d-20f1f51c4583" (UID: "e2a31c27-ece2-4af3-865d-20f1f51c4583"). InnerVolumeSpecName "kube-api-access-99dgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:21.421045 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:21.421011 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dgl\" (UniqueName: \"kubernetes.io/projected/e2a31c27-ece2-4af3-865d-20f1f51c4583-kube-api-access-99dgl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.067939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.067901 5815 generic.go:356] "Generic (PLEG): container finished" podID="e2a31c27-ece2-4af3-865d-20f1f51c4583" containerID="7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793" exitCode=137 Nov 03 21:01:22.068401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.067949 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:22.068401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.068005 5815 scope.go:122] "RemoveContainer" containerID="7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793" Nov 03 21:01:22.070186 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.070161 5815 generic.go:356] "Generic (PLEG): container finished" podID="a0ab6af4-cfb9-4054-a616-174cc898d9b0" containerID="6ab2b647702b654d2090d9ba37661c2fac31481ad5a8bea53b1af0242dfe7030" exitCode=0 Nov 03 21:01:22.070310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.070236 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" event={"ID":"a0ab6af4-cfb9-4054-a616-174cc898d9b0","Type":"ContainerDied","Data":"6ab2b647702b654d2090d9ba37661c2fac31481ad5a8bea53b1af0242dfe7030"} Nov 03 21:01:22.079421 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.079404 5815 scope.go:122] "RemoveContainer" containerID="7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793" Nov 03 21:01:22.079719 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:22.079699 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793\": container with ID starting with 7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793 not found: ID does not exist" containerID="7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793" Nov 03 21:01:22.079804 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.079728 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793"} err="failed to get container status \"7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793\": rpc error: code = NotFound desc = could not find container \"7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793\": container with ID starting with 7f3f812be34a95f1154fe33343e620486618a392838dadb2f627226c36a1a793 not found: ID does not exist" Nov 03 21:01:22.091042 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:22.091011 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-5618ac\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="e2a31c27-ece2-4af3-865d-20f1f51c4583" pod="opendatahub/test-curl-7573-5618ac" Nov 03 21:01:22.586881 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:22.586848 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a31c27-ece2-4af3-865d-20f1f51c4583" path="/var/lib/kubelet/pods/e2a31c27-ece2-4af3-865d-20f1f51c4583/volumes" Nov 03 21:01:23.336115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.336077 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:23.338185 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.338162 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:23.351683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.351660 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk\" (UniqueName: \"kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk\") pod \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " Nov 03 21:01:23.351815 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.351693 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp\" (UniqueName: \"kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp\") pod \"20358686-575e-4f17-a545-7e12558b1559\" (UID: \"20358686-575e-4f17-a545-7e12558b1559\") " Nov 03 21:01:23.351815 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.351751 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca\") pod \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " Nov 03 21:01:23.351947 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.351913 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca\") pod \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\" (UID: \"a0ab6af4-cfb9-4054-a616-174cc898d9b0\") " Nov 03 21:01:23.352361 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.352323 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca" pod "a0ab6af4-cfb9-4054-a616-174cc898d9b0" (UID: "a0ab6af4-cfb9-4054-a616-174cc898d9b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:23.352495 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.352478 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.355289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.355252 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk" pod "a0ab6af4-cfb9-4054-a616-174cc898d9b0" (UID: "a0ab6af4-cfb9-4054-a616-174cc898d9b0"). InnerVolumeSpecName "kube-api-access-675bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:23.355954 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.355914 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca" pod "a0ab6af4-cfb9-4054-a616-174cc898d9b0" (UID: "a0ab6af4-cfb9-4054-a616-174cc898d9b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:23.356307 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.356274 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp" pod "20358686-575e-4f17-a545-7e12558b1559" (UID: "20358686-575e-4f17-a545-7e12558b1559"). InnerVolumeSpecName "kube-api-access-rttmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:23.454030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.453996 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ab6af4-cfb9-4054-a616-174cc898d9b0-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.454030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.454027 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-675bk\" (UniqueName: \"kubernetes.io/projected/a0ab6af4-cfb9-4054-a616-174cc898d9b0-kube-api-access-675bk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.454030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.454038 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rttmp\" (UniqueName: \"kubernetes.io/projected/20358686-575e-4f17-a545-7e12558b1559-kube-api-access-rttmp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.488163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.488130 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 21:01:23.554688 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554649 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.554901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554707 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.554901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554782 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.554901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554863 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.554901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554896 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.555168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554931 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.555168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554951 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:23.555168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.554971 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.555168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.555030 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.555168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.555086 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token\") pod \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\" (UID: \"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1\") " Nov 03 21:01:23.555561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.555419 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:23.555872 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.555840 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:23.556003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.555982 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.556082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.556012 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.556082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.556026 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.556208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.556089 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:23.556208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.556144 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:23.557550 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.557493 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:23.557772 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.557747 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "kube-api-access-mr7tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:23.558060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.558031 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:23.558918 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.558895 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo" pod "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" (UID: "38dfcfbf-8392-4b60-a96a-7427ee3ad9d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:01:23.657331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.657241 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr7tf\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-kube-api-access-mr7tf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.657331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.657272 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.657331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.657285 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.657331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.657297 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.657331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.657309 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:23.657331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:23.657321 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:24.091442 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.091410 5815 generic.go:356] "Generic (PLEG): container finished" podID="20358686-575e-4f17-a545-7e12558b1559" containerID="aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4" exitCode=137 Nov 03 21:01:24.091663 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.091571 5815 scope.go:122] "RemoveContainer" containerID="aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4" Nov 03 21:01:24.091832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.091800 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:24.093548 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.093490 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" Nov 03 21:01:24.093548 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.093534 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k" event={"ID":"a0ab6af4-cfb9-4054-a616-174cc898d9b0","Type":"ContainerDied","Data":"0cf1136e1eedacd9e8ac1d52ead1b0c234e43483c97e9bf713162d445e473e6d"} Nov 03 21:01:24.093757 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.093577 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cf1136e1eedacd9e8ac1d52ead1b0c234e43483c97e9bf713162d445e473e6d" Nov 03 21:01:24.099462 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.099433 5815 generic.go:356] "Generic (PLEG): container finished" podID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerID="62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951" exitCode=0 Nov 03 21:01:24.099626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.099479 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" Nov 03 21:01:24.099626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.099535 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" event={"ID":"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1","Type":"ContainerDied","Data":"62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951"} Nov 03 21:01:24.099626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.099589 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl" event={"ID":"38dfcfbf-8392-4b60-a96a-7427ee3ad9d1","Type":"ContainerDied","Data":"7eb662f3064977de9dd0bc95a80d6bfd9359a475755841c1041172207c098a56"} Nov 03 21:01:24.106245 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.106222 5815 scope.go:122] "RemoveContainer" containerID="aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4" Nov 03 21:01:24.106605 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:24.106578 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4\": container with ID starting with aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4 not found: ID does not exist" containerID="aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4" Nov 03 21:01:24.106696 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.106620 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4"} err="failed to get container status \"aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4\": rpc error: code = NotFound desc = could not find container \"aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4\": container with ID starting with aaed31759e85a4c0ed9e2b006952bf1c7b87339df23a2be299f9498ed8a83fa4 not found: ID does not exist" Nov 03 21:01:24.106696 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.106644 5815 scope.go:122] "RemoveContainer" containerID="62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951" Nov 03 21:01:24.112851 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:24.112814 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-d81fc0\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="20358686-575e-4f17-a545-7e12558b1559" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:24.119642 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:24.119613 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-d81fc0\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="20358686-575e-4f17-a545-7e12558b1559" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:24.120212 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.120193 5815 scope.go:122] "RemoveContainer" containerID="ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798" Nov 03 21:01:24.130567 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.130547 5815 scope.go:122] "RemoveContainer" containerID="62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951" Nov 03 21:01:24.130863 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:24.130843 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951\": container with ID starting with 62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951 not found: ID does not exist" containerID="62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951" Nov 03 21:01:24.130946 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.130871 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951"} err="failed to get container status \"62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951\": rpc error: code = NotFound desc = could not find container \"62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951\": container with ID starting with 62dca309592032c284c08cd46c36bbb9b1a0f51b11952db34141e7e70be1f951 not found: ID does not exist" Nov 03 21:01:24.130946 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.130887 5815 scope.go:122] "RemoveContainer" containerID="ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798" Nov 03 21:01:24.131166 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:24.131149 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798\": container with ID starting with ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798 not found: ID does not exist" containerID="ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798" Nov 03 21:01:24.131224 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.131171 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798"} err="failed to get container status \"ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798\": rpc error: code = NotFound desc = could not find container \"ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798\": container with ID starting with ab3b83832ad06ddc53a8ebba327c496eed1ff5ac9ea15b2f02a4dc1a9b32f798 not found: ID does not exist" Nov 03 21:01:24.136493 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.136472 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl"] Nov 03 21:01:24.140891 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:24.140866 5815 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7573-d81fc0\" is forbidden: User \"system:node:ip-10-0-143-81.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-81.us-east-2.compute.internal' and this object" podUID="20358686-575e-4f17-a545-7e12558b1559" pod="opendatahub/test-curl-7573-d81fc0" Nov 03 21:01:24.142436 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.142416 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-mkxsl"] Nov 03 21:01:24.587118 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.587085 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20358686-575e-4f17-a545-7e12558b1559" path="/var/lib/kubelet/pods/20358686-575e-4f17-a545-7e12558b1559/volumes" Nov 03 21:01:24.587614 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:24.587595 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" path="/var/lib/kubelet/pods/38dfcfbf-8392-4b60-a96a-7427ee3ad9d1/volumes" Nov 03 21:01:29.506455 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.506427 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 21:01:29.557162 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.557132 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl\" (UniqueName: \"kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl\") pod \"5bacd210-dadc-449f-8f13-228fe7cecc3c\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " Nov 03 21:01:29.557358 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.557189 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume\") pod \"5bacd210-dadc-449f-8f13-228fe7cecc3c\" (UID: \"5bacd210-dadc-449f-8f13-228fe7cecc3c\") " Nov 03 21:01:29.557635 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.557609 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume" pod "5bacd210-dadc-449f-8f13-228fe7cecc3c" (UID: "5bacd210-dadc-449f-8f13-228fe7cecc3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:29.557755 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.557738 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bacd210-dadc-449f-8f13-228fe7cecc3c-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:29.559695 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.559665 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl" pod "5bacd210-dadc-449f-8f13-228fe7cecc3c" (UID: "5bacd210-dadc-449f-8f13-228fe7cecc3c"). InnerVolumeSpecName "kube-api-access-cldkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:29.663805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:29.663763 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cldkl\" (UniqueName: \"kubernetes.io/projected/5bacd210-dadc-449f-8f13-228fe7cecc3c-kube-api-access-cldkl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:30.149961 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.149924 5815 generic.go:356] "Generic (PLEG): container finished" podID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerID="81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07" exitCode=0 Nov 03 21:01:30.150175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.149978 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" Nov 03 21:01:30.150175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.150005 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" event={"ID":"5bacd210-dadc-449f-8f13-228fe7cecc3c","Type":"ContainerDied","Data":"81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07"} Nov 03 21:01:30.150175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.150057 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh" event={"ID":"5bacd210-dadc-449f-8f13-228fe7cecc3c","Type":"ContainerDied","Data":"12f2b6ad58c9529fd475ef012d38b9b281e438366c2993e28176c26a2421c554"} Nov 03 21:01:30.150175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.150079 5815 scope.go:122] "RemoveContainer" containerID="81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07" Nov 03 21:01:30.164007 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.163988 5815 scope.go:122] "RemoveContainer" containerID="81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07" Nov 03 21:01:30.164307 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:30.164284 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07\": container with ID starting with 81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07 not found: ID does not exist" containerID="81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07" Nov 03 21:01:30.164433 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.164313 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07"} err="failed to get container status \"81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07\": rpc error: code = NotFound desc = could not find container \"81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07\": container with ID starting with 81dd845931749f3d8b0f8bbe68f9ade6b697c3a122932ef9f240246d2c4dcc07 not found: ID does not exist" Nov 03 21:01:30.179079 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.179033 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh"] Nov 03 21:01:30.184613 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.184594 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-05473b-5bb56bc4ff-jn7hh"] Nov 03 21:01:30.566162 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566126 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq"] Nov 03 21:01:30.566786 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566768 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a0ab6af4-cfb9-4054-a616-174cc898d9b0" containerName="revoke-keys" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566786 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ab6af4-cfb9-4054-a616-174cc898d9b0" containerName="revoke-keys" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566801 5815 cpu_manager.go:414] "containerMap: removing container" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566805 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566816 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerName="payload-pre-processing" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566821 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerName="payload-pre-processing" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566835 5815 cpu_manager.go:414] "containerMap: removing container" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" Nov 03 21:01:30.566860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566840 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" Nov 03 21:01:30.567250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566966 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ab6af4-cfb9-4054-a616-174cc898d9b0" containerName="revoke-keys" Nov 03 21:01:30.567250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566976 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" Nov 03 21:01:30.567250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566983 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" containerName="payload-pre-processing" Nov 03 21:01:30.567250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.566990 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="38dfcfbf-8392-4b60-a96a-7427ee3ad9d1" containerName="istio-proxy" Nov 03 21:01:30.572930 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.572902 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.581541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.581480 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:01:30.582429 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.582395 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:30.603655 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.603621 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bacd210-dadc-449f-8f13-228fe7cecc3c" path="/var/lib/kubelet/pods/5bacd210-dadc-449f-8f13-228fe7cecc3c/volumes" Nov 03 21:01:30.604080 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.604059 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq"] Nov 03 21:01:30.682146 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.682114 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rfb\" (UniqueName: \"kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.682146 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.682159 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.682376 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.682290 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.783560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.783526 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.783798 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.783779 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rfb\" (UniqueName: \"kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.783853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.783837 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.784176 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.784157 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.786541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.786479 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.798813 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.798793 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rfb\" (UniqueName: \"kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb\") pod \"maas-api-revoke-sub-3582fe58-xlxcq\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:30.906653 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.906587 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:01:30.913893 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:30.913875 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:31.214965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:31.214934 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq"] Nov 03 21:01:31.216684 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:01:31.216657 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25adebeb_57a1_4c4f_9f92_a594dce7c388.slice/crio-27939932965d2e937a5b4bc46f46d33a7f6183f7a6744eb4e4f89c8b0cdd6b38 WatchSource:0}: Error finding container 27939932965d2e937a5b4bc46f46d33a7f6183f7a6744eb4e4f89c8b0cdd6b38: Status 404 returned error can't find the container with id 27939932965d2e937a5b4bc46f46d33a7f6183f7a6744eb4e4f89c8b0cdd6b38 Nov 03 21:01:32.171600 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:32.171563 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" event={"ID":"25adebeb-57a1-4c4f-9f92-a594dce7c388","Type":"ContainerStarted","Data":"34e6adcb43a00252bab794797a7d18e3ab8de9a2b541f4b73ad1b5ec7303f4b7"} Nov 03 21:01:32.172059 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:32.171632 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" event={"ID":"25adebeb-57a1-4c4f-9f92-a594dce7c388","Type":"ContainerStarted","Data":"27939932965d2e937a5b4bc46f46d33a7f6183f7a6744eb4e4f89c8b0cdd6b38"} Nov 03 21:01:32.190993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:32.190943 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" podStartSLOduration=2.190927117 podStartE2EDuration="2.190927117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:32.190217626 +0000 UTC m=+3601.720563299" watchObservedRunningTime="2026-11-03 21:01:32.190927117 +0000 UTC m=+3601.721272789" Nov 03 21:01:33.185376 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:33.185342 5815 generic.go:356] "Generic (PLEG): container finished" podID="25adebeb-57a1-4c4f-9f92-a594dce7c388" containerID="34e6adcb43a00252bab794797a7d18e3ab8de9a2b541f4b73ad1b5ec7303f4b7" exitCode=0 Nov 03 21:01:33.185892 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:33.185434 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" event={"ID":"25adebeb-57a1-4c4f-9f92-a594dce7c388","Type":"ContainerDied","Data":"34e6adcb43a00252bab794797a7d18e3ab8de9a2b541f4b73ad1b5ec7303f4b7"} Nov 03 21:01:34.363161 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.363135 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:34.460851 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.460826 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb\" (UniqueName: \"kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb\") pod \"25adebeb-57a1-4c4f-9f92-a594dce7c388\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " Nov 03 21:01:34.461013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.460980 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca\") pod \"25adebeb-57a1-4c4f-9f92-a594dce7c388\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " Nov 03 21:01:34.461100 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.461086 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca\") pod \"25adebeb-57a1-4c4f-9f92-a594dce7c388\" (UID: \"25adebeb-57a1-4c4f-9f92-a594dce7c388\") " Nov 03 21:01:34.461363 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.461334 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca" pod "25adebeb-57a1-4c4f-9f92-a594dce7c388" (UID: "25adebeb-57a1-4c4f-9f92-a594dce7c388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:34.461542 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.461428 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:34.463249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.463230 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb" pod "25adebeb-57a1-4c4f-9f92-a594dce7c388" (UID: "25adebeb-57a1-4c4f-9f92-a594dce7c388"). InnerVolumeSpecName "kube-api-access-h8rfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:34.463314 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.463290 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca" pod "25adebeb-57a1-4c4f-9f92-a594dce7c388" (UID: "25adebeb-57a1-4c4f-9f92-a594dce7c388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:34.563542 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.563493 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25adebeb-57a1-4c4f-9f92-a594dce7c388-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:34.563542 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:34.563546 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rfb\" (UniqueName: \"kubernetes.io/projected/25adebeb-57a1-4c4f-9f92-a594dce7c388-kube-api-access-h8rfb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:35.205434 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:35.205392 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" event={"ID":"25adebeb-57a1-4c4f-9f92-a594dce7c388","Type":"ContainerDied","Data":"27939932965d2e937a5b4bc46f46d33a7f6183f7a6744eb4e4f89c8b0cdd6b38"} Nov 03 21:01:35.205434 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:35.205426 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq" Nov 03 21:01:35.205778 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:35.205464 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27939932965d2e937a5b4bc46f46d33a7f6183f7a6744eb4e4f89c8b0cdd6b38" Nov 03 21:01:36.207152 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:36.207119 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:01:36.221418 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:36.221397 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:01:36.250578 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:36.250556 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:01:36.255979 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:36.255962 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:01:41.146854 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.146780 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l"] Nov 03 21:01:41.148784 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.148752 5815 cpu_manager.go:414] "containerMap: removing container" podUID="25adebeb-57a1-4c4f-9f92-a594dce7c388" containerName="revoke-keys" Nov 03 21:01:41.148784 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.148782 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25adebeb-57a1-4c4f-9f92-a594dce7c388" containerName="revoke-keys" Nov 03 21:01:41.149113 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.149094 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="25adebeb-57a1-4c4f-9f92-a594dce7c388" containerName="revoke-keys" Nov 03 21:01:41.161240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.155025 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.161240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.160070 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:01:41.161240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.160117 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:41.162680 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.162652 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l"] Nov 03 21:01:41.269479 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.269431 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczxt\" (UniqueName: \"kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.269781 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.269758 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.270082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.270064 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.371881 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.371838 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.372093 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.371972 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczxt\" (UniqueName: \"kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.372093 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.371996 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.372578 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.372552 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.374596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.374574 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.384799 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.384776 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczxt\" (UniqueName: \"kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt\") pod \"maas-api-revoke-sub-495a507d-gwf4l\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.524165 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.524136 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:41.811487 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:01:41.811453 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341dda15_c4d4_43cf_baef_bdc4a3cfc6d1.slice/crio-af8661a5e6d96103509e9ec96a7e1c7d5b4b8ea48b0c8583a9ca813ae44f6ae8 WatchSource:0}: Error finding container af8661a5e6d96103509e9ec96a7e1c7d5b4b8ea48b0c8583a9ca813ae44f6ae8: Status 404 returned error can't find the container with id af8661a5e6d96103509e9ec96a7e1c7d5b4b8ea48b0c8583a9ca813ae44f6ae8 Nov 03 21:01:41.812542 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:41.812499 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l"] Nov 03 21:01:42.314928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:42.314888 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" event={"ID":"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1","Type":"ContainerStarted","Data":"8133df8cd11b2c55cdebd14b88cc654dc4a163578e1475940ffc3a261dee839e"} Nov 03 21:01:42.315343 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:42.314952 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" event={"ID":"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1","Type":"ContainerStarted","Data":"af8661a5e6d96103509e9ec96a7e1c7d5b4b8ea48b0c8583a9ca813ae44f6ae8"} Nov 03 21:01:42.337188 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:42.337099 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" podStartSLOduration=1.337085697 podStartE2EDuration="1.337085697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:42.332974963 +0000 UTC m=+3611.863320635" watchObservedRunningTime="2026-11-03 21:01:42.337085697 +0000 UTC m=+3611.867431368" Nov 03 21:01:43.337524 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:43.332472 5815 generic.go:356] "Generic (PLEG): container finished" podID="341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" containerID="8133df8cd11b2c55cdebd14b88cc654dc4a163578e1475940ffc3a261dee839e" exitCode=0 Nov 03 21:01:43.337524 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:43.332615 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" event={"ID":"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1","Type":"ContainerDied","Data":"8133df8cd11b2c55cdebd14b88cc654dc4a163578e1475940ffc3a261dee839e"} Nov 03 21:01:43.853153 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:01:43.853109 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:01:44.508306 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.508278 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:44.630274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.630204 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca\") pod \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " Nov 03 21:01:44.630555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.630343 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt\" (UniqueName: \"kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt\") pod \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " Nov 03 21:01:44.630555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.630427 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca\") pod \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\" (UID: \"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1\") " Nov 03 21:01:44.630774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.630613 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca" pod "341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" (UID: "341dda15-c4d4-43cf-baef-bdc4a3cfc6d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:44.631029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.631011 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:44.632716 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.632688 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt" pod "341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" (UID: "341dda15-c4d4-43cf-baef-bdc4a3cfc6d1"). InnerVolumeSpecName "kube-api-access-gczxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:44.632817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.632691 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca" pod "341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" (UID: "341dda15-c4d4-43cf-baef-bdc4a3cfc6d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:44.732456 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.732423 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczxt\" (UniqueName: \"kubernetes.io/projected/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-kube-api-access-gczxt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:44.732456 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:44.732453 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:45.351673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:45.351644 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" Nov 03 21:01:45.351863 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:45.351675 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l" event={"ID":"341dda15-c4d4-43cf-baef-bdc4a3cfc6d1","Type":"ContainerDied","Data":"af8661a5e6d96103509e9ec96a7e1c7d5b4b8ea48b0c8583a9ca813ae44f6ae8"} Nov 03 21:01:45.351863 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:45.351745 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af8661a5e6d96103509e9ec96a7e1c7d5b4b8ea48b0c8583a9ca813ae44f6ae8" Nov 03 21:01:46.838705 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:46.838672 5815 scope.go:122] "RemoveContainer" containerID="44f483be7a55e6418c8ed40bfc92fb539c718684bc8dbd1a74f4407a666564df" Nov 03 21:01:46.850459 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:46.850435 5815 scope.go:122] "RemoveContainer" containerID="64826a85d28ac634c21a7fc06e8fbff827a7de2ac5a49f24f176d9ce869a8863" Nov 03 21:01:51.525612 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.525575 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v"] Nov 03 21:01:51.533055 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.531412 5815 cpu_manager.go:414] "containerMap: removing container" podUID="341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" containerName="revoke-keys" Nov 03 21:01:51.533337 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.533309 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" containerName="revoke-keys" Nov 03 21:01:51.533775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.533753 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" containerName="revoke-keys" Nov 03 21:01:51.543660 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.543632 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v"] Nov 03 21:01:51.543822 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.543736 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.549118 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.549091 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:51.549252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.549148 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:01:51.591193 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.591169 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625c2\" (UniqueName: \"kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.591303 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.591284 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.591574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.591557 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.692390 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.692364 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625c2\" (UniqueName: \"kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.692555 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.692460 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.692640 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.692578 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.693386 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.693358 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.695047 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.695025 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.704607 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.704582 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625c2\" (UniqueName: \"kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2\") pod \"maas-api-revoke-sub-31489283-hgq7v\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:51.860176 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:51.860100 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:52.148291 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:52.148264 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v"] Nov 03 21:01:52.148756 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:01:52.148732 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51770a0b_05ca_424f_8bbd_0cda586850a4.slice/crio-f20e940f244c2c2b5d11778ac39b42b3ec0509e9b78804da0fd3bb48236ca3a6 WatchSource:0}: Error finding container f20e940f244c2c2b5d11778ac39b42b3ec0509e9b78804da0fd3bb48236ca3a6: Status 404 returned error can't find the container with id f20e940f244c2c2b5d11778ac39b42b3ec0509e9b78804da0fd3bb48236ca3a6 Nov 03 21:01:52.499758 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:52.499709 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" event={"ID":"51770a0b-05ca-424f-8bbd-0cda586850a4","Type":"ContainerStarted","Data":"f20e940f244c2c2b5d11778ac39b42b3ec0509e9b78804da0fd3bb48236ca3a6"} Nov 03 21:01:53.511412 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:53.511369 5815 generic.go:356] "Generic (PLEG): container finished" podID="51770a0b-05ca-424f-8bbd-0cda586850a4" containerID="23fe75d42ba9e08b34c6bf00495bdf2088f399b81424d682879b78af97dae2cd" exitCode=0 Nov 03 21:01:53.511830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:53.511441 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" event={"ID":"51770a0b-05ca-424f-8bbd-0cda586850a4","Type":"ContainerDied","Data":"23fe75d42ba9e08b34c6bf00495bdf2088f399b81424d682879b78af97dae2cd"} Nov 03 21:01:54.719712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.719683 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:01:54.844021 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.843937 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca\") pod \"51770a0b-05ca-424f-8bbd-0cda586850a4\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " Nov 03 21:01:54.844021 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.843988 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca\") pod \"51770a0b-05ca-424f-8bbd-0cda586850a4\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " Nov 03 21:01:54.844249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.844107 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2\" (UniqueName: \"kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2\") pod \"51770a0b-05ca-424f-8bbd-0cda586850a4\" (UID: \"51770a0b-05ca-424f-8bbd-0cda586850a4\") " Nov 03 21:01:54.844350 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.844322 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca" pod "51770a0b-05ca-424f-8bbd-0cda586850a4" (UID: "51770a0b-05ca-424f-8bbd-0cda586850a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:54.844947 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.844922 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:54.846693 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.846655 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca" pod "51770a0b-05ca-424f-8bbd-0cda586850a4" (UID: "51770a0b-05ca-424f-8bbd-0cda586850a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:54.847111 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.847086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2" pod "51770a0b-05ca-424f-8bbd-0cda586850a4" (UID: "51770a0b-05ca-424f-8bbd-0cda586850a4"). InnerVolumeSpecName "kube-api-access-625c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:54.946267 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.946241 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51770a0b-05ca-424f-8bbd-0cda586850a4-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:54.946378 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:54.946268 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625c2\" (UniqueName: \"kubernetes.io/projected/51770a0b-05ca-424f-8bbd-0cda586850a4-kube-api-access-625c2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:55.528487 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:55.528453 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" event={"ID":"51770a0b-05ca-424f-8bbd-0cda586850a4","Type":"ContainerDied","Data":"f20e940f244c2c2b5d11778ac39b42b3ec0509e9b78804da0fd3bb48236ca3a6"} Nov 03 21:01:55.528699 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:55.528500 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f20e940f244c2c2b5d11778ac39b42b3ec0509e9b78804da0fd3bb48236ca3a6" Nov 03 21:01:55.528699 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:01:55.528458 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v" Nov 03 21:02:00.155214 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:00.155182 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:02:01.710734 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:01.710699 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb"] Nov 03 21:02:01.711668 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:01.711632 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" containerID="cri-o://8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2" gracePeriod=30 Nov 03 21:02:02.173425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.173333 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5"] Nov 03 21:02:02.599737 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.599699 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx"] Nov 03 21:02:02.600379 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.600357 5815 cpu_manager.go:414] "containerMap: removing container" podUID="51770a0b-05ca-424f-8bbd-0cda586850a4" containerName="revoke-keys" Nov 03 21:02:02.600545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.600380 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51770a0b-05ca-424f-8bbd-0cda586850a4" containerName="revoke-keys" Nov 03 21:02:02.600624 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.600576 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="51770a0b-05ca-424f-8bbd-0cda586850a4" containerName="revoke-keys" Nov 03 21:02:02.604175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.604155 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.610901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.610880 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:02.611227 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.611206 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:02:02.613610 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.613587 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx"] Nov 03 21:02:02.709845 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.709809 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.710028 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.709920 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcsf7\" (UniqueName: \"kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.710173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.710154 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.811803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.811766 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcsf7\" (UniqueName: \"kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.812305 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.811899 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.812305 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.812011 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.812787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.812764 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.814665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.814641 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.824372 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.824350 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcsf7\" (UniqueName: \"kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7\") pod \"maas-api-revoke-sub-a9e5cdca-bcshx\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:02.942704 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.942215 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77"] Nov 03 21:02:02.942704 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.942436 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" containerID="cri-o://0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc" gracePeriod=30 Nov 03 21:02:02.946174 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.946126 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr"] Nov 03 21:02:02.946389 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.946343 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" containerID="cri-o://c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0" gracePeriod=30 Nov 03 21:02:02.950799 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.950768 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g"] Nov 03 21:02:02.951031 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.950989 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" containerID="cri-o://bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65" gracePeriod=30 Nov 03 21:02:02.959453 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:02.959427 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:03.288184 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:03.288154 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx"] Nov 03 21:02:03.288340 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:02:03.288312 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077a476d_da55_4fd8_8f91_62d84f5a4d34.slice/crio-fc2ee1bde490d172e3c78e5e5d4258d0c2ba4db46320944a46967de3de703bd1 WatchSource:0}: Error finding container fc2ee1bde490d172e3c78e5e5d4258d0c2ba4db46320944a46967de3de703bd1: Status 404 returned error can't find the container with id fc2ee1bde490d172e3c78e5e5d4258d0c2ba4db46320944a46967de3de703bd1 Nov 03 21:02:03.606839 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:03.606804 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" event={"ID":"077a476d-da55-4fd8-8f91-62d84f5a4d34","Type":"ContainerStarted","Data":"fc2ee1bde490d172e3c78e5e5d4258d0c2ba4db46320944a46967de3de703bd1"} Nov 03 21:02:04.623405 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:04.623315 5815 generic.go:356] "Generic (PLEG): container finished" podID="077a476d-da55-4fd8-8f91-62d84f5a4d34" containerID="b48299085189931ac8b539cc8de8146cc10b8149a03e9c54dd14afb04f1e9cad" exitCode=0 Nov 03 21:02:04.624220 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:04.624191 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" event={"ID":"077a476d-da55-4fd8-8f91-62d84f5a4d34","Type":"ContainerDied","Data":"b48299085189931ac8b539cc8de8146cc10b8149a03e9c54dd14afb04f1e9cad"} Nov 03 21:02:04.798810 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:04.798095 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj"] Nov 03 21:02:05.797091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.795979 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:05.961534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.961484 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7\" (UniqueName: \"kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7\") pod \"077a476d-da55-4fd8-8f91-62d84f5a4d34\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " Nov 03 21:02:05.961742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.961638 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca\") pod \"077a476d-da55-4fd8-8f91-62d84f5a4d34\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " Nov 03 21:02:05.961830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.961770 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca\") pod \"077a476d-da55-4fd8-8f91-62d84f5a4d34\" (UID: \"077a476d-da55-4fd8-8f91-62d84f5a4d34\") " Nov 03 21:02:05.962400 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.962367 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca" pod "077a476d-da55-4fd8-8f91-62d84f5a4d34" (UID: "077a476d-da55-4fd8-8f91-62d84f5a4d34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:05.964012 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.963973 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca" pod "077a476d-da55-4fd8-8f91-62d84f5a4d34" (UID: "077a476d-da55-4fd8-8f91-62d84f5a4d34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:05.964161 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:05.964105 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7" pod "077a476d-da55-4fd8-8f91-62d84f5a4d34" (UID: "077a476d-da55-4fd8-8f91-62d84f5a4d34"). InnerVolumeSpecName "kube-api-access-vcsf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:06.063806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.063778 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcsf7\" (UniqueName: \"kubernetes.io/projected/077a476d-da55-4fd8-8f91-62d84f5a4d34-kube-api-access-vcsf7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:06.063806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.063806 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:06.064005 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.063816 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077a476d-da55-4fd8-8f91-62d84f5a4d34-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:06.351460 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.351377 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf"] Nov 03 21:02:06.352130 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.352112 5815 cpu_manager.go:414] "containerMap: removing container" podUID="077a476d-da55-4fd8-8f91-62d84f5a4d34" containerName="revoke-keys" Nov 03 21:02:06.352192 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.352132 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077a476d-da55-4fd8-8f91-62d84f5a4d34" containerName="revoke-keys" Nov 03 21:02:06.352247 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.352238 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="077a476d-da55-4fd8-8f91-62d84f5a4d34" containerName="revoke-keys" Nov 03 21:02:06.355884 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.355861 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.365684 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.365601 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf"] Nov 03 21:02:06.468563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.468529 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzf9k\" (UniqueName: \"kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.468766 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.468738 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.468896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.468871 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.570381 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.570345 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzf9k\" (UniqueName: \"kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.570637 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.570444 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.570637 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.570490 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.571188 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.571160 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.573283 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.573260 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.587213 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.587179 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzf9k\" (UniqueName: \"kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k\") pod \"maas-api-revoke-sub-abd08420-cdzjf\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.640968 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.640905 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" Nov 03 21:02:06.641097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.640953 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx" event={"ID":"077a476d-da55-4fd8-8f91-62d84f5a4d34","Type":"ContainerDied","Data":"fc2ee1bde490d172e3c78e5e5d4258d0c2ba4db46320944a46967de3de703bd1"} Nov 03 21:02:06.641097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.641019 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2ee1bde490d172e3c78e5e5d4258d0c2ba4db46320944a46967de3de703bd1" Nov 03 21:02:06.682736 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.682690 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-dtftr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:02:06.682855 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.682748 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:02:06.684901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.684887 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:06.970873 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:06.970844 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf"] Nov 03 21:02:06.989428 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:02:06.989393 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e43a07b_30d8_42ef_ae78_36a2f37ea6d1.slice/crio-1b71212d03bb3f11219b8a4b050108a41d782204ff727cc24f1904f0524f7940 WatchSource:0}: Error finding container 1b71212d03bb3f11219b8a4b050108a41d782204ff727cc24f1904f0524f7940: Status 404 returned error can't find the container with id 1b71212d03bb3f11219b8a4b050108a41d782204ff727cc24f1904f0524f7940 Nov 03 21:02:07.184008 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.183978 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd"] Nov 03 21:02:07.276009 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.275984 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 21:02:07.384160 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384129 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384197 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384292 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384342 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384378 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384405 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384474 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:07.384626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384617 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384607 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:07.384993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384708 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:07.384993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384711 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384777 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token\") pod \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\" (UID: \"65d4bf87-dd67-4905-bf8f-bb093e231c4d\") " Nov 03 21:02:07.384993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.384865 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:07.385368 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.385041 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:07.385444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.385412 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.385533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.385449 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.385533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.385467 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.385533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.385482 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.385533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.385496 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.387991 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.387951 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:07.388441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.388421 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:07.388618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.388595 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:07.388940 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.388919 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd" pod "65d4bf87-dd67-4905-bf8f-bb093e231c4d" (UID: "65d4bf87-dd67-4905-bf8f-bb093e231c4d"). InnerVolumeSpecName "kube-api-access-4z2kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:07.486885 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.486856 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.486885 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.486883 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.486885 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.486893 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2kd\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-kube-api-access-4z2kd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.487127 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.486904 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65d4bf87-dd67-4905-bf8f-bb093e231c4d-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:07.650636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.650552 5815 generic.go:356] "Generic (PLEG): container finished" podID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerID="8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2" exitCode=0 Nov 03 21:02:07.650636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.650585 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" Nov 03 21:02:07.650636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.650611 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" event={"ID":"65d4bf87-dd67-4905-bf8f-bb093e231c4d","Type":"ContainerDied","Data":"8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2"} Nov 03 21:02:07.650945 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.650680 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb" event={"ID":"65d4bf87-dd67-4905-bf8f-bb093e231c4d","Type":"ContainerDied","Data":"5c5986c65cd789ca1e18b1e540371fec087e6491c24d5e6d5db6d8a6d8a87f3b"} Nov 03 21:02:07.650945 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.650702 5815 scope.go:122] "RemoveContainer" containerID="8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2" Nov 03 21:02:07.653207 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.653177 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" event={"ID":"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1","Type":"ContainerStarted","Data":"0e0cb0a4e57d6e58ec02171f6eed1a0944fb4d6eedb523e86fa2f3991465ddeb"} Nov 03 21:02:07.653316 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.653241 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" event={"ID":"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1","Type":"ContainerStarted","Data":"1b71212d03bb3f11219b8a4b050108a41d782204ff727cc24f1904f0524f7940"} Nov 03 21:02:07.663317 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.663300 5815 scope.go:122] "RemoveContainer" containerID="838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380" Nov 03 21:02:07.674630 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.674587 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" podStartSLOduration=1.674574367 podStartE2EDuration="1.674574367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:07.673680126 +0000 UTC m=+3637.204025799" watchObservedRunningTime="2026-11-03 21:02:07.674574367 +0000 UTC m=+3637.204920051" Nov 03 21:02:07.679136 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.679109 5815 scope.go:122] "RemoveContainer" containerID="8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2" Nov 03 21:02:07.679459 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:07.679438 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2\": container with ID starting with 8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2 not found: ID does not exist" containerID="8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2" Nov 03 21:02:07.679592 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.679467 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2"} err="failed to get container status \"8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2\": rpc error: code = NotFound desc = could not find container \"8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2\": container with ID starting with 8f944f2605542a733c307e5db1efbd58edc1dad9d1da5c8d6ae9d282a6eeaeb2 not found: ID does not exist" Nov 03 21:02:07.679592 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.679482 5815 scope.go:122] "RemoveContainer" containerID="838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380" Nov 03 21:02:07.679838 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:07.679784 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380\": container with ID starting with 838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380 not found: ID does not exist" containerID="838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380" Nov 03 21:02:07.680460 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.679846 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380"} err="failed to get container status \"838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380\": rpc error: code = NotFound desc = could not find container \"838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380\": container with ID starting with 838b496474874c5108cbe3aae0fbbe445bef823c43545c6c14f23d5a314b3380 not found: ID does not exist" Nov 03 21:02:07.692671 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.692297 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb"] Nov 03 21:02:07.712129 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.712100 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-08a7bc-openshift-default-5864477cc-5k4kb"] Nov 03 21:02:07.907459 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.907393 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-ksr77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:02:07.907670 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:07.907458 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:02:08.243556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.240107 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 21:02:08.322834 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.322808 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 21:02:08.326294 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.326210 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402208 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402267 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402312 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402348 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402439 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402476 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402501 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.402625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402559 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.403060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.402645 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket\") pod \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\" (UID: \"d28af1a7-45a3-4ed5-895c-b8370f9d0496\") " Nov 03 21:02:08.403206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.403176 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.403442 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.403420 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.404081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.404051 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.404194 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.404167 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:08.404424 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.404359 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.406088 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.406058 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.406197 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.406114 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:08.406570 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.406548 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "kube-api-access-l7g2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.406659 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.406635 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token" pod "d28af1a7-45a3-4ed5-895c-b8370f9d0496" (UID: "d28af1a7-45a3-4ed5-895c-b8370f9d0496"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.504092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.503999 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504040 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504086 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504170 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504187 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504212 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504231 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504296 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504325 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504329 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.504385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504363 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504396 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504448 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504482 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504530 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data\") pod \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\" (UID: \"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504573 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504597 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504638 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504652 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504708 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504781 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.504908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504849 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo\") pod \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\" (UID: \"600d178b-1ec4-458a-b05c-468ad5d6a1f9\") " Nov 03 21:02:08.505575 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.504935 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.505575 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505056 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:08.505575 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505324 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.505575 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505495 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505870 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505901 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505966 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.505984 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506001 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506017 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506034 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506050 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506065 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506081 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506096 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506113 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7g2j\" (UniqueName: \"kubernetes.io/projected/d28af1a7-45a3-4ed5-895c-b8370f9d0496-kube-api-access-l7g2j\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506130 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.506147 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506145 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.507089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506161 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.507089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506176 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.507089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506191 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.507089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506208 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d28af1a7-45a3-4ed5-895c-b8370f9d0496-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.507089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.506400 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.507430 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.507414 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:08.507486 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.507440 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.507563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.507484 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "kube-api-access-kfnvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.507867 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.507833 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:08.508001 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.507978 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:08.508769 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.508748 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token" pod "600d178b-1ec4-458a-b05c-468ad5d6a1f9" (UID: "600d178b-1ec4-458a-b05c-468ad5d6a1f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.509714 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.509693 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.509785 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.509728 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9" pod "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" (UID: "81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6"). InnerVolumeSpecName "kube-api-access-9mck9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.586953 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.586919 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" path="/var/lib/kubelet/pods/65d4bf87-dd67-4905-bf8f-bb093e231c4d/volumes" Nov 03 21:02:08.607368 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607339 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mck9\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-kube-api-access-9mck9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607373 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607383 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607401 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607411 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfnvc\" (UniqueName: \"kubernetes.io/projected/600d178b-1ec4-458a-b05c-468ad5d6a1f9-kube-api-access-kfnvc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607421 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607430 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607441 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/600d178b-1ec4-458a-b05c-468ad5d6a1f9-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607450 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.607482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.607459 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.662690 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.662633 5815 generic.go:356] "Generic (PLEG): container finished" podID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerID="0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc" exitCode=0 Nov 03 21:02:08.662690 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.662675 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" Nov 03 21:02:08.662870 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.662722 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" event={"ID":"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6","Type":"ContainerDied","Data":"0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc"} Nov 03 21:02:08.662870 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.662788 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77" event={"ID":"81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6","Type":"ContainerDied","Data":"471d2143297ddc0344d38bb4235f1cfb5feb2a55543cf7e9900abe29580bdd93"} Nov 03 21:02:08.662870 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.662825 5815 scope.go:122] "RemoveContainer" containerID="0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc" Nov 03 21:02:08.665141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.665110 5815 generic.go:356] "Generic (PLEG): container finished" podID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerID="c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0" exitCode=0 Nov 03 21:02:08.665314 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.665299 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" Nov 03 21:02:08.666044 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.665948 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" event={"ID":"d28af1a7-45a3-4ed5-895c-b8370f9d0496","Type":"ContainerDied","Data":"c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0"} Nov 03 21:02:08.666044 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.666000 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr" event={"ID":"d28af1a7-45a3-4ed5-895c-b8370f9d0496","Type":"ContainerDied","Data":"757b90b50c8f0a493d1a7c94f34cd996a6a4cb831f89070bb393d5bdb8d2b9b8"} Nov 03 21:02:08.671501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.671327 5815 generic.go:356] "Generic (PLEG): container finished" podID="8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" containerID="0e0cb0a4e57d6e58ec02171f6eed1a0944fb4d6eedb523e86fa2f3991465ddeb" exitCode=0 Nov 03 21:02:08.671501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.671453 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" event={"ID":"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1","Type":"ContainerDied","Data":"0e0cb0a4e57d6e58ec02171f6eed1a0944fb4d6eedb523e86fa2f3991465ddeb"} Nov 03 21:02:08.673230 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.673206 5815 generic.go:356] "Generic (PLEG): container finished" podID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerID="bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65" exitCode=0 Nov 03 21:02:08.673342 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.673326 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" Nov 03 21:02:08.673425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.673410 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" event={"ID":"600d178b-1ec4-458a-b05c-468ad5d6a1f9","Type":"ContainerDied","Data":"bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65"} Nov 03 21:02:08.673477 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.673459 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g" event={"ID":"600d178b-1ec4-458a-b05c-468ad5d6a1f9","Type":"ContainerDied","Data":"cc87c4f233321f2ad1653b90cdf73a279a1b26aec4276ea4d9f5af642e8b8ae6"} Nov 03 21:02:08.677702 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.677680 5815 scope.go:122] "RemoveContainer" containerID="0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc" Nov 03 21:02:08.678034 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:08.678009 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc\": container with ID starting with 0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc not found: ID does not exist" containerID="0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc" Nov 03 21:02:08.678140 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.678041 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc"} err="failed to get container status \"0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc\": rpc error: code = NotFound desc = could not find container \"0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc\": container with ID starting with 0586bb62734652af4ac5062adc56440ba5f97318d9df92f2e3021e7842c910bc not found: ID does not exist" Nov 03 21:02:08.678140 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.678059 5815 scope.go:122] "RemoveContainer" containerID="c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0" Nov 03 21:02:08.689397 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.689366 5815 scope.go:122] "RemoveContainer" containerID="c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0" Nov 03 21:02:08.689750 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:08.689723 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0\": container with ID starting with c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0 not found: ID does not exist" containerID="c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0" Nov 03 21:02:08.689850 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.689759 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0"} err="failed to get container status \"c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0\": rpc error: code = NotFound desc = could not find container \"c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0\": container with ID starting with c9386625d77df856a547bd7ea089bcf9fe534403304b6e752268df7f1bb097e0 not found: ID does not exist" Nov 03 21:02:08.689850 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.689782 5815 scope.go:122] "RemoveContainer" containerID="bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65" Nov 03 21:02:08.694011 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.693991 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77"] Nov 03 21:02:08.700994 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.700976 5815 scope.go:122] "RemoveContainer" containerID="bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65" Nov 03 21:02:08.701256 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:08.701238 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65\": container with ID starting with bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65 not found: ID does not exist" containerID="bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65" Nov 03 21:02:08.701330 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.701262 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65"} err="failed to get container status \"bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65\": rpc error: code = NotFound desc = could not find container \"bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65\": container with ID starting with bd041fe8b533c73975161099daa6a6ba86e0f7333b7bd1826d6dba30b201be65 not found: ID does not exist" Nov 03 21:02:08.702415 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.702396 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-ksr77"] Nov 03 21:02:08.738876 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.738853 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr"] Nov 03 21:02:08.744698 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.744675 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-dtftr"] Nov 03 21:02:08.758714 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.758638 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g"] Nov 03 21:02:08.763333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:08.763311 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-pt86g"] Nov 03 21:02:09.844012 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.843985 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:09.927424 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.927390 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k\" (UniqueName: \"kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k\") pod \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " Nov 03 21:02:09.927631 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.927622 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca\") pod \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " Nov 03 21:02:09.927683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.927643 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca\") pod \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\" (UID: \"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1\") " Nov 03 21:02:09.928040 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.928013 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca" pod "8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" (UID: "8e43a07b-30d8-42ef-ae78-36a2f37ea6d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:09.929904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.929869 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k" pod "8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" (UID: "8e43a07b-30d8-42ef-ae78-36a2f37ea6d1"). InnerVolumeSpecName "kube-api-access-wzf9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:09.930006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:09.929916 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca" pod "8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" (UID: "8e43a07b-30d8-42ef-ae78-36a2f37ea6d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:10.028965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.028932 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzf9k\" (UniqueName: \"kubernetes.io/projected/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-kube-api-access-wzf9k\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.028965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.028965 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.029173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.028976 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.588728 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.588692 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" path="/var/lib/kubelet/pods/600d178b-1ec4-458a-b05c-468ad5d6a1f9/volumes" Nov 03 21:02:10.589189 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.589175 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" path="/var/lib/kubelet/pods/81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6/volumes" Nov 03 21:02:10.589640 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.589626 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" path="/var/lib/kubelet/pods/d28af1a7-45a3-4ed5-895c-b8370f9d0496/volumes" Nov 03 21:02:10.696356 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.696321 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" Nov 03 21:02:10.696571 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.696423 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf" event={"ID":"8e43a07b-30d8-42ef-ae78-36a2f37ea6d1","Type":"ContainerDied","Data":"1b71212d03bb3f11219b8a4b050108a41d782204ff727cc24f1904f0524f7940"} Nov 03 21:02:10.696571 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:10.696471 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b71212d03bb3f11219b8a4b050108a41d782204ff727cc24f1904f0524f7940" Nov 03 21:02:13.941780 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:13.941617 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:02:13.952453 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.952425 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2"] Nov 03 21:02:13.953031 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953017 5815 cpu_manager.go:414] "containerMap: removing container" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953032 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953046 5815 cpu_manager.go:414] "containerMap: removing container" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953050 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953068 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953073 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953080 5815 cpu_manager.go:414] "containerMap: removing container" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" Nov 03 21:02:13.953081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953084 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953105 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" containerName="revoke-keys" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953110 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" containerName="revoke-keys" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953124 5815 cpu_manager.go:414] "containerMap: removing container" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953129 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953213 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28af1a7-45a3-4ed5-895c-b8370f9d0496" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953223 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e4bdbb-1ab2-4419-845c-1eb5d9e0dfa6" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953232 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" containerName="revoke-keys" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953240 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="600d178b-1ec4-458a-b05c-468ad5d6a1f9" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953247 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" Nov 03 21:02:13.953310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.953252 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d4bf87-dd67-4905-bf8f-bb093e231c4d" containerName="istio-proxy" Nov 03 21:02:13.958098 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.958081 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:13.968531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.962235 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:13.968531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.962344 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:02:13.968531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:13.966327 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2"] Nov 03 21:02:14.082354 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.082320 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.082354 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.082357 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.082678 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.082520 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplls\" (UniqueName: \"kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.184451 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.184418 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.184451 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.184456 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.184785 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.184534 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplls\" (UniqueName: \"kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.185240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.185210 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.187157 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.187133 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.197573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.197466 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplls\" (UniqueName: \"kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls\") pod \"maas-api-revoke-sub-7a73c9e9-5kqc2\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.293018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.292989 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:14.581428 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:02:14.581396 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29721775_b0a9_4845_9c2b_fe0d5c67d11e.slice/crio-15680a1205881e72d6d6d6d73c510f194cf751bf15001bed681cf76587840bb8 WatchSource:0}: Error finding container 15680a1205881e72d6d6d6d73c510f194cf751bf15001bed681cf76587840bb8: Status 404 returned error can't find the container with id 15680a1205881e72d6d6d6d73c510f194cf751bf15001bed681cf76587840bb8 Nov 03 21:02:14.589336 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.589308 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2"] Nov 03 21:02:14.730125 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:14.730087 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" event={"ID":"29721775-b0a9-4845-9c2b-fe0d5c67d11e","Type":"ContainerStarted","Data":"15680a1205881e72d6d6d6d73c510f194cf751bf15001bed681cf76587840bb8"} Nov 03 21:02:15.739973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:15.739934 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" event={"ID":"29721775-b0a9-4845-9c2b-fe0d5c67d11e","Type":"ContainerStarted","Data":"df2eccbb4a1a855f2606694eb3eb7e2f33ec60b1a9d9fdaa9b42b42e270d1d82"} Nov 03 21:02:15.760302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:15.760259 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" podStartSLOduration=2.760245235 podStartE2EDuration="2.760245235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:15.757833489 +0000 UTC m=+3645.288179163" watchObservedRunningTime="2026-11-03 21:02:15.760245235 +0000 UTC m=+3645.290590907" Nov 03 21:02:16.571324 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.571280 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz"] Nov 03 21:02:16.575055 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.575033 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.606425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.606394 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz"] Nov 03 21:02:16.618039 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.618007 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc94p\" (UniqueName: \"kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.618270 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.618087 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.618270 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.618206 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.718948 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.718912 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.719108 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.719024 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc94p\" (UniqueName: \"kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.719108 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.719065 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.719596 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.719573 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.721641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.721617 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.731221 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.731195 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc94p\" (UniqueName: \"kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p\") pod \"maas-api-revoke-sub-24787d73-vs5vz\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:16.749335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.749310 5815 generic.go:356] "Generic (PLEG): container finished" podID="29721775-b0a9-4845-9c2b-fe0d5c67d11e" containerID="df2eccbb4a1a855f2606694eb3eb7e2f33ec60b1a9d9fdaa9b42b42e270d1d82" exitCode=0 Nov 03 21:02:16.749701 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.749396 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" event={"ID":"29721775-b0a9-4845-9c2b-fe0d5c67d11e","Type":"ContainerDied","Data":"df2eccbb4a1a855f2606694eb3eb7e2f33ec60b1a9d9fdaa9b42b42e270d1d82"} Nov 03 21:02:16.905657 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:16.905563 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:17.210648 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.210617 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz"] Nov 03 21:02:17.210890 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:02:17.210862 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d6a935_afc8_4ae0_8b73_9c6d4a0c314a.slice/crio-af64bfe3dfb76d674f863137021a9de1ad09ab220f89167069c4905afd662193 WatchSource:0}: Error finding container af64bfe3dfb76d674f863137021a9de1ad09ab220f89167069c4905afd662193: Status 404 returned error can't find the container with id af64bfe3dfb76d674f863137021a9de1ad09ab220f89167069c4905afd662193 Nov 03 21:02:17.214433 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.214404 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" podUID="61d29238-aee6-4218-9234-af485c8df7c2" containerName="main" containerID="cri-o://a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab" gracePeriod=45 Nov 03 21:02:17.486666 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.486640 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 21:02:17.536187 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536160 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs\") pod \"61d29238-aee6-4218-9234-af485c8df7c2\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " Nov 03 21:02:17.536387 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536291 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7\" (UniqueName: \"kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7\") pod \"61d29238-aee6-4218-9234-af485c8df7c2\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " Nov 03 21:02:17.536387 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536337 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache\") pod \"61d29238-aee6-4218-9234-af485c8df7c2\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " Nov 03 21:02:17.536573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536386 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm\") pod \"61d29238-aee6-4218-9234-af485c8df7c2\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " Nov 03 21:02:17.536573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536433 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home\") pod \"61d29238-aee6-4218-9234-af485c8df7c2\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " Nov 03 21:02:17.536573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536491 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir\") pod \"61d29238-aee6-4218-9234-af485c8df7c2\" (UID: \"61d29238-aee6-4218-9234-af485c8df7c2\") " Nov 03 21:02:17.536758 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536685 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache" pod "61d29238-aee6-4218-9234-af485c8df7c2" (UID: "61d29238-aee6-4218-9234-af485c8df7c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:17.536758 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536696 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home" pod "61d29238-aee6-4218-9234-af485c8df7c2" (UID: "61d29238-aee6-4218-9234-af485c8df7c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:17.536983 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.536844 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir" pod "61d29238-aee6-4218-9234-af485c8df7c2" (UID: "61d29238-aee6-4218-9234-af485c8df7c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:17.537072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.537043 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:17.537072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.537065 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:17.538477 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.538451 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs" pod "61d29238-aee6-4218-9234-af485c8df7c2" (UID: "61d29238-aee6-4218-9234-af485c8df7c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:17.538634 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.538538 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm" pod "61d29238-aee6-4218-9234-af485c8df7c2" (UID: "61d29238-aee6-4218-9234-af485c8df7c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:17.538685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.538671 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7" pod "61d29238-aee6-4218-9234-af485c8df7c2" (UID: "61d29238-aee6-4218-9234-af485c8df7c2"). InnerVolumeSpecName "kube-api-access-rjsj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:17.638761 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.638736 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d29238-aee6-4218-9234-af485c8df7c2-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:17.638761 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.638761 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjsj7\" (UniqueName: \"kubernetes.io/projected/61d29238-aee6-4218-9234-af485c8df7c2-kube-api-access-rjsj7\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:17.638957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.638771 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:17.638957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.638780 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d29238-aee6-4218-9234-af485c8df7c2-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:17.758755 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.758724 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" event={"ID":"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a","Type":"ContainerStarted","Data":"5407caaa16bd2181381df64bbee9db3431c894ef87883a41eee35023d15f487e"} Nov 03 21:02:17.759217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.758780 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" event={"ID":"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a","Type":"ContainerStarted","Data":"af64bfe3dfb76d674f863137021a9de1ad09ab220f89167069c4905afd662193"} Nov 03 21:02:17.760238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.760214 5815 generic.go:356] "Generic (PLEG): container finished" podID="61d29238-aee6-4218-9234-af485c8df7c2" containerID="a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab" exitCode=0 Nov 03 21:02:17.760318 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.760249 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" Nov 03 21:02:17.760318 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.760289 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" event={"ID":"61d29238-aee6-4218-9234-af485c8df7c2","Type":"ContainerDied","Data":"a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab"} Nov 03 21:02:17.760421 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.760325 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5" event={"ID":"61d29238-aee6-4218-9234-af485c8df7c2","Type":"ContainerDied","Data":"e75dfd42004b6d82b585b8ff8393fb7132364f881526aa6d402ab1d2855edd50"} Nov 03 21:02:17.760421 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.760345 5815 scope.go:122] "RemoveContainer" containerID="a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab" Nov 03 21:02:17.774854 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.774829 5815 scope.go:122] "RemoveContainer" containerID="a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab" Nov 03 21:02:17.775170 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:17.775150 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab\": container with ID starting with a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab not found: ID does not exist" containerID="a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab" Nov 03 21:02:17.775253 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.775182 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab"} err="failed to get container status \"a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab\": rpc error: code = NotFound desc = could not find container \"a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab\": container with ID starting with a718c890571aadad3a433e99bdc1db1a73364d44de63fc433750ab68014958ab not found: ID does not exist" Nov 03 21:02:17.795061 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.795007 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" podStartSLOduration=1.794989449 podStartE2EDuration="1.794989449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:17.781090849 +0000 UTC m=+3647.311436586" watchObservedRunningTime="2026-11-03 21:02:17.794989449 +0000 UTC m=+3647.325335128" Nov 03 21:02:17.799119 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.799091 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5"] Nov 03 21:02:17.802807 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.802783 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bab2c3/sub-test-model-bab2c3-kserve-5b6dc5444c-tpgn5"] Nov 03 21:02:17.966060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:17.966029 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:18.045286 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.045203 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca\") pod \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " Nov 03 21:02:18.045463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.045306 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls\" (UniqueName: \"kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls\") pod \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " Nov 03 21:02:18.045463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.045427 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca\") pod \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\" (UID: \"29721775-b0a9-4845-9c2b-fe0d5c67d11e\") " Nov 03 21:02:18.045663 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.045636 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca" pod "29721775-b0a9-4845-9c2b-fe0d5c67d11e" (UID: "29721775-b0a9-4845-9c2b-fe0d5c67d11e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:18.046002 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.045982 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.047674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.047651 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls" pod "29721775-b0a9-4845-9c2b-fe0d5c67d11e" (UID: "29721775-b0a9-4845-9c2b-fe0d5c67d11e"). InnerVolumeSpecName "kube-api-access-dplls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:18.047774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.047688 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca" pod "29721775-b0a9-4845-9c2b-fe0d5c67d11e" (UID: "29721775-b0a9-4845-9c2b-fe0d5c67d11e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:18.147038 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.147003 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29721775-b0a9-4845-9c2b-fe0d5c67d11e-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.147038 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.147033 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplls\" (UniqueName: \"kubernetes.io/projected/29721775-b0a9-4845-9c2b-fe0d5c67d11e-kube-api-access-dplls\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.587609 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.587571 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d29238-aee6-4218-9234-af485c8df7c2" path="/var/lib/kubelet/pods/61d29238-aee6-4218-9234-af485c8df7c2/volumes" Nov 03 21:02:18.772807 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.772777 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" Nov 03 21:02:18.773284 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.772804 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2" event={"ID":"29721775-b0a9-4845-9c2b-fe0d5c67d11e","Type":"ContainerDied","Data":"15680a1205881e72d6d6d6d73c510f194cf751bf15001bed681cf76587840bb8"} Nov 03 21:02:18.773284 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.772866 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15680a1205881e72d6d6d6d73c510f194cf751bf15001bed681cf76587840bb8" Nov 03 21:02:18.776586 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.776559 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" event={"ID":"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a","Type":"ContainerDied","Data":"5407caaa16bd2181381df64bbee9db3431c894ef87883a41eee35023d15f487e"} Nov 03 21:02:18.776719 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:18.776495 5815 generic.go:356] "Generic (PLEG): container finished" podID="94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" containerID="5407caaa16bd2181381df64bbee9db3431c894ef87883a41eee35023d15f487e" exitCode=0 Nov 03 21:02:19.832942 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:19.832895 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" podUID="1f803152-09a3-4007-8a43-612b71e83a63" containerName="main" containerID="cri-o://36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7" gracePeriod=45 Nov 03 21:02:19.958258 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:19.958230 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" podUID="1f803152-09a3-4007-8a43-612b71e83a63" containerName="main" probeResult="failure" output="Get \"https://10.132.1.160:8000/ready\": dial tcp 10.132.1.160:8000: connect: connection refused" Nov 03 21:02:19.963973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:19.963952 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:20.083177 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.083089 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca\") pod \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " Nov 03 21:02:20.083340 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.083188 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca\") pod \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " Nov 03 21:02:20.083571 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.083453 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p\" (UniqueName: \"kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p\") pod \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\" (UID: \"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a\") " Nov 03 21:02:20.083910 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.083876 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca" pod "94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" (UID: "94d6a935-afc8-4ae0-8b73-9c6d4a0c314a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:20.086079 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.084274 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.091874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.091851 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca" pod "94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" (UID: "94d6a935-afc8-4ae0-8b73-9c6d4a0c314a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:20.092598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.092569 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p" pod "94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" (UID: "94d6a935-afc8-4ae0-8b73-9c6d4a0c314a"). InnerVolumeSpecName "kube-api-access-nc94p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:20.129717 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.129685 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 21:02:20.185261 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185230 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454\" (UniqueName: \"kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454\") pod \"1f803152-09a3-4007-8a43-612b71e83a63\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " Nov 03 21:02:20.185480 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185309 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache\") pod \"1f803152-09a3-4007-8a43-612b71e83a63\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " Nov 03 21:02:20.185480 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185369 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home\") pod \"1f803152-09a3-4007-8a43-612b71e83a63\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " Nov 03 21:02:20.185480 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185436 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs\") pod \"1f803152-09a3-4007-8a43-612b71e83a63\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " Nov 03 21:02:20.185693 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185564 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir\") pod \"1f803152-09a3-4007-8a43-612b71e83a63\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " Nov 03 21:02:20.185693 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185605 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache" pod "1f803152-09a3-4007-8a43-612b71e83a63" (UID: "1f803152-09a3-4007-8a43-612b71e83a63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:20.185693 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185675 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm\") pod \"1f803152-09a3-4007-8a43-612b71e83a63\" (UID: \"1f803152-09a3-4007-8a43-612b71e83a63\") " Nov 03 21:02:20.185846 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185710 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home" pod "1f803152-09a3-4007-8a43-612b71e83a63" (UID: "1f803152-09a3-4007-8a43-612b71e83a63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:20.185902 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.185882 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir" pod "1f803152-09a3-4007-8a43-612b71e83a63" (UID: "1f803152-09a3-4007-8a43-612b71e83a63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:20.186415 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.186390 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.186574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.186418 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.186574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.186435 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc94p\" (UniqueName: \"kubernetes.io/projected/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-kube-api-access-nc94p\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.186574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.186450 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.186574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.186486 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.187848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.187822 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454" pod "1f803152-09a3-4007-8a43-612b71e83a63" (UID: "1f803152-09a3-4007-8a43-612b71e83a63"). InnerVolumeSpecName "kube-api-access-d7454". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:20.188439 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.188417 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs" pod "1f803152-09a3-4007-8a43-612b71e83a63" (UID: "1f803152-09a3-4007-8a43-612b71e83a63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:20.188566 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.188422 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm" pod "1f803152-09a3-4007-8a43-612b71e83a63" (UID: "1f803152-09a3-4007-8a43-612b71e83a63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:20.287658 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.287619 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f803152-09a3-4007-8a43-612b71e83a63-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.287658 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.287658 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f803152-09a3-4007-8a43-612b71e83a63-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.287946 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.287673 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7454\" (UniqueName: \"kubernetes.io/projected/1f803152-09a3-4007-8a43-612b71e83a63-kube-api-access-d7454\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.796489 5815 generic.go:356] "Generic (PLEG): container finished" podID="1f803152-09a3-4007-8a43-612b71e83a63" containerID="36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7" exitCode=0 Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.796569 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.796607 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" event={"ID":"1f803152-09a3-4007-8a43-612b71e83a63","Type":"ContainerDied","Data":"36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7"} Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.796652 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj" event={"ID":"1f803152-09a3-4007-8a43-612b71e83a63","Type":"ContainerDied","Data":"503b82729d488b1152681706d56b05be8a4a3233a22924e91639b4bb1b97582c"} Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.796674 5815 scope.go:122] "RemoveContainer" containerID="36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7" Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.798343 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.798373 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz" event={"ID":"94d6a935-afc8-4ae0-8b73-9c6d4a0c314a","Type":"ContainerDied","Data":"af64bfe3dfb76d674f863137021a9de1ad09ab220f89167069c4905afd662193"} Nov 03 21:02:20.800545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.798424 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af64bfe3dfb76d674f863137021a9de1ad09ab220f89167069c4905afd662193" Nov 03 21:02:20.808986 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.808970 5815 scope.go:122] "RemoveContainer" containerID="36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7" Nov 03 21:02:20.809297 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:20.809277 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7\": container with ID starting with 36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7 not found: ID does not exist" containerID="36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7" Nov 03 21:02:20.809344 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.809306 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7"} err="failed to get container status \"36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7\": rpc error: code = NotFound desc = could not find container \"36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7\": container with ID starting with 36520efdec89fd41eac1b59174794624ca62a55d5d1109a1641ca1c2367b99b7 not found: ID does not exist" Nov 03 21:02:20.825420 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.825388 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj"] Nov 03 21:02:20.834060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:20.834035 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6e5139/sub-test-model-6e5139-kserve-5894d74b59-wzvmj"] Nov 03 21:02:22.208940 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.208896 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" containerID="cri-o://eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88" gracePeriod=45 Nov 03 21:02:22.602879 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.602742 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f803152-09a3-4007-8a43-612b71e83a63" path="/var/lib/kubelet/pods/1f803152-09a3-4007-8a43-612b71e83a63/volumes" Nov 03 21:02:22.640101 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.640075 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 21:02:22.738653 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738571 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache\") pod \"d87d17d2-6ce0-4368-b147-401e31d32450\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " Nov 03 21:02:22.738830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738665 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home\") pod \"d87d17d2-6ce0-4368-b147-401e31d32450\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " Nov 03 21:02:22.738830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738722 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir\") pod \"d87d17d2-6ce0-4368-b147-401e31d32450\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " Nov 03 21:02:22.738830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738759 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs\") pod \"d87d17d2-6ce0-4368-b147-401e31d32450\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " Nov 03 21:02:22.738830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738788 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h\" (UniqueName: \"kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h\") pod \"d87d17d2-6ce0-4368-b147-401e31d32450\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " Nov 03 21:02:22.738830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738820 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm\") pod \"d87d17d2-6ce0-4368-b147-401e31d32450\" (UID: \"d87d17d2-6ce0-4368-b147-401e31d32450\") " Nov 03 21:02:22.739145 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738821 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache" pod "d87d17d2-6ce0-4368-b147-401e31d32450" (UID: "d87d17d2-6ce0-4368-b147-401e31d32450"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.739145 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.738956 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home" pod "d87d17d2-6ce0-4368-b147-401e31d32450" (UID: "d87d17d2-6ce0-4368-b147-401e31d32450"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.739145 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.739049 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir" pod "d87d17d2-6ce0-4368-b147-401e31d32450" (UID: "d87d17d2-6ce0-4368-b147-401e31d32450"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.739436 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.739416 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.739546 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.739441 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.739546 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.739455 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.741407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.741384 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm" pod "d87d17d2-6ce0-4368-b147-401e31d32450" (UID: "d87d17d2-6ce0-4368-b147-401e31d32450"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.741561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.741416 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs" pod "d87d17d2-6ce0-4368-b147-401e31d32450" (UID: "d87d17d2-6ce0-4368-b147-401e31d32450"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:22.741561 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.741460 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h" pod "d87d17d2-6ce0-4368-b147-401e31d32450" (UID: "d87d17d2-6ce0-4368-b147-401e31d32450"). InnerVolumeSpecName "kube-api-access-d7k9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:22.822680 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.822635 5815 generic.go:356] "Generic (PLEG): container finished" podID="d87d17d2-6ce0-4368-b147-401e31d32450" containerID="eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88" exitCode=0 Nov 03 21:02:22.822866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.822694 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" Nov 03 21:02:22.822866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.822717 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" event={"ID":"d87d17d2-6ce0-4368-b147-401e31d32450","Type":"ContainerDied","Data":"eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88"} Nov 03 21:02:22.822866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.822765 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd" event={"ID":"d87d17d2-6ce0-4368-b147-401e31d32450","Type":"ContainerDied","Data":"dc4e5082874719d5c7abc1c1038642ff86e66a2848051da4454649442aec0120"} Nov 03 21:02:22.822866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.822782 5815 scope.go:122] "RemoveContainer" containerID="eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88" Nov 03 21:02:22.834425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.834406 5815 scope.go:122] "RemoveContainer" containerID="eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88" Nov 03 21:02:22.834742 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:22.834722 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88\": container with ID starting with eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88 not found: ID does not exist" containerID="eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88" Nov 03 21:02:22.834798 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.834750 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88"} err="failed to get container status \"eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88\": rpc error: code = NotFound desc = could not find container \"eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88\": container with ID starting with eba5c5932674dbd24d15d256ee9ff9eab9a7329716a5d95a44def7c718b8ea88 not found: ID does not exist" Nov 03 21:02:22.841101 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.841074 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d87d17d2-6ce0-4368-b147-401e31d32450-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.841101 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.841096 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7k9h\" (UniqueName: \"kubernetes.io/projected/d87d17d2-6ce0-4368-b147-401e31d32450-kube-api-access-d7k9h\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.841250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.841106 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d87d17d2-6ce0-4368-b147-401e31d32450-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.850626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.850606 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd"] Nov 03 21:02:22.855214 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:22.855195 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/premium-w1-b13a14-kserve-7d457dbbd9-m24pd"] Nov 03 21:02:24.587899 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:24.587867 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" path="/var/lib/kubelet/pods/d87d17d2-6ce0-4368-b147-401e31d32450/volumes" Nov 03 21:02:27.205844 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:27.204473 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6"] Nov 03 21:02:27.515536 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:27.513578 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh"] Nov 03 21:02:27.515536 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:27.513826 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" containerID="cri-o://d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27" gracePeriod=30 Nov 03 21:02:30.412592 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:30.412548 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl"] Nov 03 21:02:32.833989 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.833961 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 21:02:32.885429 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885392 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.885632 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885475 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.885632 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885581 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.885774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885733 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.885835 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885792 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.885895 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885853 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.885895 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885880 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.886003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885918 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.886003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.885987 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz\") pod \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\" (UID: \"ac6976b5-e1d9-442b-9b8f-a06b6edf290b\") " Nov 03 21:02:32.886390 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.886365 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:32.887304 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.887258 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:32.888079 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.888052 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:32.888199 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.888152 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:32.888478 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.888457 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:32.891241 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.890618 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "kube-api-access-xvlfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:32.891241 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.890654 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:32.891759 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.891732 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:32.892756 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.892726 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo" pod "ac6976b5-e1d9-442b-9b8f-a06b6edf290b" (UID: "ac6976b5-e1d9-442b-9b8f-a06b6edf290b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:32.927728 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.927695 5815 generic.go:356] "Generic (PLEG): container finished" podID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerID="d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27" exitCode=0 Nov 03 21:02:32.927874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.927759 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" Nov 03 21:02:32.927874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.927766 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" event={"ID":"ac6976b5-e1d9-442b-9b8f-a06b6edf290b","Type":"ContainerDied","Data":"d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27"} Nov 03 21:02:32.927874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.927815 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh" event={"ID":"ac6976b5-e1d9-442b-9b8f-a06b6edf290b","Type":"ContainerDied","Data":"ce465c75b442a963baba24df8ed367713dccd1a738a3f02ba8527c5baaaafceb"} Nov 03 21:02:32.927874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.927842 5815 scope.go:122] "RemoveContainer" containerID="d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27" Nov 03 21:02:32.940055 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.940038 5815 scope.go:122] "RemoveContainer" containerID="00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264" Nov 03 21:02:32.949996 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.949975 5815 scope.go:122] "RemoveContainer" containerID="d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27" Nov 03 21:02:32.950352 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:32.950312 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27\": container with ID starting with d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27 not found: ID does not exist" containerID="d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27" Nov 03 21:02:32.950459 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.950347 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27"} err="failed to get container status \"d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27\": rpc error: code = NotFound desc = could not find container \"d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27\": container with ID starting with d7e2b87cda1040189531ff4e9c71e5a478be30ed82e31e5f3847bfb1830d0f27 not found: ID does not exist" Nov 03 21:02:32.950459 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.950366 5815 scope.go:122] "RemoveContainer" containerID="00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264" Nov 03 21:02:32.950836 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:32.950780 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264\": container with ID starting with 00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264 not found: ID does not exist" containerID="00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264" Nov 03 21:02:32.950930 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.950843 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264"} err="failed to get container status \"00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264\": rpc error: code = NotFound desc = could not find container \"00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264\": container with ID starting with 00386962926fe5af256e2998a9208516a5101e6856cc32f321dc079825559264 not found: ID does not exist" Nov 03 21:02:32.955107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.955086 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh"] Nov 03 21:02:32.958885 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.958866 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-14ebbb-openshift-default-56b94b45bf-jtjkh"] Nov 03 21:02:32.987255 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987233 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987255 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987255 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987267 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987277 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987286 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987296 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlfz\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-kube-api-access-xvlfz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987304 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987313 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:32.987388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:32.987329 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac6976b5-e1d9-442b-9b8f-a06b6edf290b-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:34.575391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:34.575360 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6"] Nov 03 21:02:34.610782 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:34.609202 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" path="/var/lib/kubelet/pods/ac6976b5-e1d9-442b-9b8f-a06b6edf290b/volumes" Nov 03 21:02:38.684807 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.684769 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc"] Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685620 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685644 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685661 5815 cpu_manager.go:414] "containerMap: removing container" podUID="94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" containerName="revoke-keys" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685671 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" containerName="revoke-keys" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685695 5815 cpu_manager.go:414] "containerMap: removing container" podUID="61d29238-aee6-4218-9234-af485c8df7c2" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685703 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d29238-aee6-4218-9234-af485c8df7c2" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685718 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1f803152-09a3-4007-8a43-612b71e83a63" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685727 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f803152-09a3-4007-8a43-612b71e83a63" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685760 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685767 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685785 5815 cpu_manager.go:414] "containerMap: removing container" podUID="29721775-b0a9-4845-9c2b-fe0d5c67d11e" containerName="revoke-keys" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685792 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29721775-b0a9-4845-9c2b-fe0d5c67d11e" containerName="revoke-keys" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685815 5815 cpu_manager.go:414] "containerMap: removing container" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685821 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685941 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" containerName="revoke-keys" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685954 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d29238-aee6-4218-9234-af485c8df7c2" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685964 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f803152-09a3-4007-8a43-612b71e83a63" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685978 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685988 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="29721775-b0a9-4845-9c2b-fe0d5c67d11e" containerName="revoke-keys" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.685999 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87d17d2-6ce0-4368-b147-401e31d32450" containerName="main" Nov 03 21:02:38.686123 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.686010 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6976b5-e1d9-442b-9b8f-a06b6edf290b" containerName="istio-proxy" Nov 03 21:02:38.692049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.692027 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.717575 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.717547 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc"] Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.824670 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.824828 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.824913 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.824947 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.824979 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjw97\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.825009 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.825050 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.825086 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.836585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.825131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926379 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926336 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926426 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926451 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926479 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjw97\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926536 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926578 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.926633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926615 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927026 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926657 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927026 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.926700 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927161 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.927087 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927266 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.927244 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927341 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.927242 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927341 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.927306 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.927450 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.927394 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.929954 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.929922 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.930139 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.930118 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.938700 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.938673 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:38.938825 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:38.938771 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjw97\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:39.024272 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.024245 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:39.322707 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.322676 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc"] Nov 03 21:02:39.324003 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:02:39.323979 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b300e92_11db_4ff9_b4fa_35fe3665d322.slice/crio-79d5bdb11aa5ab24c3664f709729aee5e8a474d4f22093bfffab41b9e648ae99 WatchSource:0}: Error finding container 79d5bdb11aa5ab24c3664f709729aee5e8a474d4f22093bfffab41b9e648ae99: Status 404 returned error can't find the container with id 79d5bdb11aa5ab24c3664f709729aee5e8a474d4f22093bfffab41b9e648ae99 Nov 03 21:02:39.326963 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.326924 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:02:39.327053 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.327039 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:02:39.327107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.327090 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:02:39.576044 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.575981 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" podUID="c226212b-ca3d-4b73-8ed5-719fe27c463d" containerName="payload-processing" containerID="cri-o://3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7" gracePeriod=25 Nov 03 21:02:39.853083 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.853060 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 21:02:39.936362 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.936323 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9\" (UniqueName: \"kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9\") pod \"c226212b-ca3d-4b73-8ed5-719fe27c463d\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " Nov 03 21:02:39.936585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.936540 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume\") pod \"c226212b-ca3d-4b73-8ed5-719fe27c463d\" (UID: \"c226212b-ca3d-4b73-8ed5-719fe27c463d\") " Nov 03 21:02:39.937115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.937086 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume" pod "c226212b-ca3d-4b73-8ed5-719fe27c463d" (UID: "c226212b-ca3d-4b73-8ed5-719fe27c463d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:39.937493 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.937476 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c226212b-ca3d-4b73-8ed5-719fe27c463d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:39.938877 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.938859 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9" pod "c226212b-ca3d-4b73-8ed5-719fe27c463d" (UID: "c226212b-ca3d-4b73-8ed5-719fe27c463d"). InnerVolumeSpecName "kube-api-access-z47g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:39.996775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.996738 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" event={"ID":"0b300e92-11db-4ff9-b4fa-35fe3665d322","Type":"ContainerStarted","Data":"a3eb74e06640d93552b2cdf121b7259c7e0af267a25adbdce289d8c7359f00c0"} Nov 03 21:02:39.997266 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.996844 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" event={"ID":"0b300e92-11db-4ff9-b4fa-35fe3665d322","Type":"ContainerStarted","Data":"79d5bdb11aa5ab24c3664f709729aee5e8a474d4f22093bfffab41b9e648ae99"} Nov 03 21:02:39.998693 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.998666 5815 generic.go:356] "Generic (PLEG): container finished" podID="c226212b-ca3d-4b73-8ed5-719fe27c463d" containerID="3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7" exitCode=0 Nov 03 21:02:39.998819 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.998713 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" Nov 03 21:02:39.998819 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.998756 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" event={"ID":"c226212b-ca3d-4b73-8ed5-719fe27c463d","Type":"ContainerDied","Data":"3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7"} Nov 03 21:02:39.998819 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.998802 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6" event={"ID":"c226212b-ca3d-4b73-8ed5-719fe27c463d","Type":"ContainerDied","Data":"61cad69aeed77893e17147b2118530d6c52215d1c3384b5173a1309a84dafb88"} Nov 03 21:02:39.998975 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:39.998824 5815 scope.go:122] "RemoveContainer" containerID="3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7" Nov 03 21:02:40.013062 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.013046 5815 scope.go:122] "RemoveContainer" containerID="3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7" Nov 03 21:02:40.013347 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:40.013328 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7\": container with ID starting with 3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7 not found: ID does not exist" containerID="3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7" Nov 03 21:02:40.013429 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.013361 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7"} err="failed to get container status \"3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7\": rpc error: code = NotFound desc = could not find container \"3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7\": container with ID starting with 3ed7b3572d2fe32ee4c20a6d401cc76e2df5b45ed772fe86e5b757ee297c38f7 not found: ID does not exist" Nov 03 21:02:40.021452 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.021397 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podStartSLOduration=2.021383181 podStartE2EDuration="2.021383181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:40.019775393 +0000 UTC m=+3669.550121087" watchObservedRunningTime="2026-11-03 21:02:40.021383181 +0000 UTC m=+3669.551728852" Nov 03 21:02:40.024367 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.024349 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:40.025699 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.025672 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:40.025768 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.025714 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:40.033481 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.033452 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72"] Nov 03 21:02:40.039164 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.039142 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z47g9\" (UniqueName: \"kubernetes.io/projected/c226212b-ca3d-4b73-8ed5-719fe27c463d-kube-api-access-z47g9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:40.040809 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.040787 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d3c979-qql72"] Nov 03 21:02:40.046571 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.046550 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6"] Nov 03 21:02:40.051931 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.051906 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6350b9-5c98f9f596-58rw6"] Nov 03 21:02:40.605534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.604538 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1e89ab-2973-4f78-ab89-942ac86351b0" path="/var/lib/kubelet/pods/8e1e89ab-2973-4f78-ab89-942ac86351b0/volumes" Nov 03 21:02:40.605534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:40.605212 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c226212b-ca3d-4b73-8ed5-719fe27c463d" path="/var/lib/kubelet/pods/c226212b-ca3d-4b73-8ed5-719fe27c463d/volumes" Nov 03 21:02:41.025072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:41.025034 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:41.025497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:41.025092 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:42.025157 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.025122 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:42.025635 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.025169 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:42.231126 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.231080 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" podUID="278413d0-ca91-4575-8120-a1514610eaf4" containerName="main" containerID="cri-o://d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e" gracePeriod=45 Nov 03 21:02:42.605019 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.604952 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 21:02:42.701594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.699973 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs\" (UniqueName: \"kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs\") pod \"278413d0-ca91-4575-8120-a1514610eaf4\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " Nov 03 21:02:42.701594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.700449 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm\") pod \"278413d0-ca91-4575-8120-a1514610eaf4\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " Nov 03 21:02:42.701594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.700858 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw\" (UniqueName: \"kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw\") pod \"278413d0-ca91-4575-8120-a1514610eaf4\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " Nov 03 21:02:42.701594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.700989 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache\") pod \"278413d0-ca91-4575-8120-a1514610eaf4\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " Nov 03 21:02:42.701594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.701081 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home\") pod \"278413d0-ca91-4575-8120-a1514610eaf4\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " Nov 03 21:02:42.701594 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.701123 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir\") pod \"278413d0-ca91-4575-8120-a1514610eaf4\" (UID: \"278413d0-ca91-4575-8120-a1514610eaf4\") " Nov 03 21:02:42.702121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.701762 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir" pod "278413d0-ca91-4575-8120-a1514610eaf4" (UID: "278413d0-ca91-4575-8120-a1514610eaf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:42.702121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.701787 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache" pod "278413d0-ca91-4575-8120-a1514610eaf4" (UID: "278413d0-ca91-4575-8120-a1514610eaf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:42.702121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.701862 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home" pod "278413d0-ca91-4575-8120-a1514610eaf4" (UID: "278413d0-ca91-4575-8120-a1514610eaf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:42.704248 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.704209 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw" pod "278413d0-ca91-4575-8120-a1514610eaf4" (UID: "278413d0-ca91-4575-8120-a1514610eaf4"). InnerVolumeSpecName "kube-api-access-r8gxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:42.704656 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.704615 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs" pod "278413d0-ca91-4575-8120-a1514610eaf4" (UID: "278413d0-ca91-4575-8120-a1514610eaf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:42.706212 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.706188 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm" pod "278413d0-ca91-4575-8120-a1514610eaf4" (UID: "278413d0-ca91-4575-8120-a1514610eaf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:42.805674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.805544 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:42.805674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.805584 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:42.805674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.805599 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:42.805674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.805611 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/278413d0-ca91-4575-8120-a1514610eaf4-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:42.805674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.805623 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/278413d0-ca91-4575-8120-a1514610eaf4-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:42.805674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:42.805639 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8gxw\" (UniqueName: \"kubernetes.io/projected/278413d0-ca91-4575-8120-a1514610eaf4-kube-api-access-r8gxw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:43.025134 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.024627 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:43.025134 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.024689 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:43.036054 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.036015 5815 generic.go:356] "Generic (PLEG): container finished" podID="278413d0-ca91-4575-8120-a1514610eaf4" containerID="d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e" exitCode=0 Nov 03 21:02:43.036324 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.036301 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" event={"ID":"278413d0-ca91-4575-8120-a1514610eaf4","Type":"ContainerDied","Data":"d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e"} Nov 03 21:02:43.036441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.036345 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" event={"ID":"278413d0-ca91-4575-8120-a1514610eaf4","Type":"ContainerDied","Data":"78a40f4b4616f4a1dc9cbfcba9a760e050c48108ce27242d0137fc9ff39a2a9f"} Nov 03 21:02:43.036441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.036368 5815 scope.go:122] "RemoveContainer" containerID="d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e" Nov 03 21:02:43.036598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.036550 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6" Nov 03 21:02:43.058730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.054780 5815 scope.go:122] "RemoveContainer" containerID="d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e" Nov 03 21:02:43.058730 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:43.055112 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e\": container with ID starting with d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e not found: ID does not exist" containerID="d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e" Nov 03 21:02:43.058730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.055195 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e"} err="failed to get container status \"d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e\": rpc error: code = NotFound desc = could not find container \"d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e\": container with ID starting with d2727d1c6c2ff9fa2608b05491d3611e176984327ce59ba838d1507bc707547e not found: ID does not exist" Nov 03 21:02:43.116665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.116618 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6"] Nov 03 21:02:43.118447 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:43.118421 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-63fde4/rate-test-model-63fde4-kserve-6ccb477c-fcgt6"] Nov 03 21:02:43.945731 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:43.945589 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:02:44.025450 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:44.024930 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:44.025450 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:44.025029 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:44.253252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:44.253222 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:02:44.625665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:44.625574 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278413d0-ca91-4575-8120-a1514610eaf4" path="/var/lib/kubelet/pods/278413d0-ca91-4575-8120-a1514610eaf4/volumes" Nov 03 21:02:45.024922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.024881 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:45.025134 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.024951 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:45.447192 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.447069 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" podUID="70c47ff1-8f40-4082-bb80-b2752b747dfd" containerName="main" containerID="cri-o://1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2" gracePeriod=45 Nov 03 21:02:45.836907 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.836554 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 21:02:45.933622 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.933590 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl\" (UniqueName: \"kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl\") pod \"70c47ff1-8f40-4082-bb80-b2752b747dfd\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " Nov 03 21:02:45.933801 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.933731 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache\") pod \"70c47ff1-8f40-4082-bb80-b2752b747dfd\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " Nov 03 21:02:45.933891 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.933815 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm\") pod \"70c47ff1-8f40-4082-bb80-b2752b747dfd\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " Nov 03 21:02:45.933962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.933899 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs\" (UniqueName: \"kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs\") pod \"70c47ff1-8f40-4082-bb80-b2752b747dfd\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " Nov 03 21:02:45.934044 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.934029 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home\") pod \"70c47ff1-8f40-4082-bb80-b2752b747dfd\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " Nov 03 21:02:45.934114 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.934076 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir\") pod \"70c47ff1-8f40-4082-bb80-b2752b747dfd\" (UID: \"70c47ff1-8f40-4082-bb80-b2752b747dfd\") " Nov 03 21:02:45.934446 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.934400 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache" pod "70c47ff1-8f40-4082-bb80-b2752b747dfd" (UID: "70c47ff1-8f40-4082-bb80-b2752b747dfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:45.934636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.934580 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:45.934724 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.934649 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir" pod "70c47ff1-8f40-4082-bb80-b2752b747dfd" (UID: "70c47ff1-8f40-4082-bb80-b2752b747dfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:45.934855 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.934835 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home" pod "70c47ff1-8f40-4082-bb80-b2752b747dfd" (UID: "70c47ff1-8f40-4082-bb80-b2752b747dfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:45.936805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.936779 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs" pod "70c47ff1-8f40-4082-bb80-b2752b747dfd" (UID: "70c47ff1-8f40-4082-bb80-b2752b747dfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:45.937085 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.936992 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm" pod "70c47ff1-8f40-4082-bb80-b2752b747dfd" (UID: "70c47ff1-8f40-4082-bb80-b2752b747dfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:45.937085 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:45.937051 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl" pod "70c47ff1-8f40-4082-bb80-b2752b747dfd" (UID: "70c47ff1-8f40-4082-bb80-b2752b747dfd"). InnerVolumeSpecName "kube-api-access-2hnvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:46.025638 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.025554 5815 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" start-of-body= Nov 03 21:02:46.025638 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.025624 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.194:15021/healthz/ready\": dial tcp 10.132.1.194:15021: connect: connection refused" Nov 03 21:02:46.035295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.035264 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:46.035295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.035294 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hnvl\" (UniqueName: \"kubernetes.io/projected/70c47ff1-8f40-4082-bb80-b2752b747dfd-kube-api-access-2hnvl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:46.035469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.035306 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:46.035469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.035318 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70c47ff1-8f40-4082-bb80-b2752b747dfd-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:46.035469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.035329 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70c47ff1-8f40-4082-bb80-b2752b747dfd-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:46.098328 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.098294 5815 generic.go:356] "Generic (PLEG): container finished" podID="70c47ff1-8f40-4082-bb80-b2752b747dfd" containerID="1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2" exitCode=0 Nov 03 21:02:46.098497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.098376 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" Nov 03 21:02:46.098497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.098405 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" event={"ID":"70c47ff1-8f40-4082-bb80-b2752b747dfd","Type":"ContainerDied","Data":"1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2"} Nov 03 21:02:46.098497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.098449 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl" event={"ID":"70c47ff1-8f40-4082-bb80-b2752b747dfd","Type":"ContainerDied","Data":"a5284037aeb3647ed3ef213375e76760ac4f4568669beb29b47882c466412906"} Nov 03 21:02:46.098497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.098468 5815 scope.go:122] "RemoveContainer" containerID="1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2" Nov 03 21:02:46.112644 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.112457 5815 scope.go:122] "RemoveContainer" containerID="1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2" Nov 03 21:02:46.112907 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:46.112884 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2\": container with ID starting with 1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2 not found: ID does not exist" containerID="1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2" Nov 03 21:02:46.113016 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.112914 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2"} err="failed to get container status \"1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2\": rpc error: code = NotFound desc = could not find container \"1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2\": container with ID starting with 1e64c4ae5ac42fb406b60dd8b909af18211edfd67747949e57511e615f1595a2 not found: ID does not exist" Nov 03 21:02:46.125955 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.125929 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl"] Nov 03 21:02:46.134229 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.134210 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-3bcc2b/rate-test-model-3bcc2b-kserve-68c8b5c68f-99rkl"] Nov 03 21:02:46.585826 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.585795 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c47ff1-8f40-4082-bb80-b2752b747dfd" path="/var/lib/kubelet/pods/70c47ff1-8f40-4082-bb80-b2752b747dfd/volumes" Nov 03 21:02:46.936713 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:46.936630 5815 scope.go:122] "RemoveContainer" containerID="d30d952b928ca3828274c30868f11e9ea0186b08bc509a3c59af06e1670fa64b" Nov 03 21:02:47.181327 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:47.181269 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:47.181757 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:47.181727 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:47.182558 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:47.182539 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:48.264847 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:48.264818 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v"] Nov 03 21:02:48.265331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:48.265041 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" containerID="cri-o://5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a" gracePeriod=30 Nov 03 21:02:48.274408 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:48.274377 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc"] Nov 03 21:02:49.070955 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:49.070912 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd"] Nov 03 21:02:49.141265 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:49.141230 5815 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" secret="" err="secret \"e2e-worker-w1-b13a14-openshift-default-dockercfg-r6zzp\" not found" Nov 03 21:02:50.148680 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:50.148646 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" containerID="cri-o://a3eb74e06640d93552b2cdf121b7259c7e0af267a25adbdce289d8c7359f00c0" gracePeriod=30 Nov 03 21:02:53.539731 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.539678 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 21:02:53.643030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.642997 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643063 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643100 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643132 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643202 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643247 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643311 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643338 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643332 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.643444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643407 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm\") pod \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\" (UID: \"1eaf8a64-6829-4034-89f8-a2608c2b7cc0\") " Nov 03 21:02:53.643771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643602 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:53.643771 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643644 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.643914 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.643889 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.644124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.644063 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.644124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.644094 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.644124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.644109 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.644124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.644125 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.644474 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.644209 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.645798 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.645772 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.645930 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.645909 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "kube-api-access-lc5fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:53.646069 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.646053 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:53.646144 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.646106 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo" pod "1eaf8a64-6829-4034-89f8-a2608c2b7cc0" (UID: "1eaf8a64-6829-4034-89f8-a2608c2b7cc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:53.745520 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.745480 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5fm\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-kube-api-access-lc5fm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.745520 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.745526 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.745795 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.745539 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.745795 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.745548 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.745795 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:53.745556 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eaf8a64-6829-4034-89f8-a2608c2b7cc0-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:54.071449 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.071400 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" podUID="f7cb8fbc-4f0b-4288-a297-f8ce9d066613" containerName="payload-processing" containerID="cri-o://8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293" gracePeriod=25 Nov 03 21:02:54.182017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.181982 5815 generic.go:356] "Generic (PLEG): container finished" podID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerID="5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a" exitCode=0 Nov 03 21:02:54.182198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.182053 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" event={"ID":"1eaf8a64-6829-4034-89f8-a2608c2b7cc0","Type":"ContainerDied","Data":"5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a"} Nov 03 21:02:54.182198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.182109 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" Nov 03 21:02:54.182198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.182123 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v" event={"ID":"1eaf8a64-6829-4034-89f8-a2608c2b7cc0","Type":"ContainerDied","Data":"4fbd328281a0dadadc2bd0fa1d53eb6ce6cb986b3124a197d29b5a5975c21967"} Nov 03 21:02:54.182198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.182153 5815 scope.go:122] "RemoveContainer" containerID="5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a" Nov 03 21:02:54.195451 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.195429 5815 scope.go:122] "RemoveContainer" containerID="5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a" Nov 03 21:02:54.195841 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:54.195816 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a\": container with ID starting with 5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a not found: ID does not exist" containerID="5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a" Nov 03 21:02:54.195949 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.195848 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a"} err="failed to get container status \"5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a\": rpc error: code = NotFound desc = could not find container \"5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a\": container with ID starting with 5cf925f6371996359511f51f771e2a0b9be26dde3ad13a6e2147e66771af973a not found: ID does not exist" Nov 03 21:02:54.209770 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.209741 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v"] Nov 03 21:02:54.213533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.213493 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-bfg8v"] Nov 03 21:02:54.360330 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.360299 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 21:02:54.454636 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.454607 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd\" (UniqueName: \"kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd\") pod \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " Nov 03 21:02:54.454833 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.454815 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume\") pod \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\" (UID: \"f7cb8fbc-4f0b-4288-a297-f8ce9d066613\") " Nov 03 21:02:54.455193 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.455166 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume" pod "f7cb8fbc-4f0b-4288-a297-f8ce9d066613" (UID: "f7cb8fbc-4f0b-4288-a297-f8ce9d066613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:54.457521 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.457474 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd" pod "f7cb8fbc-4f0b-4288-a297-f8ce9d066613" (UID: "f7cb8fbc-4f0b-4288-a297-f8ce9d066613"). InnerVolumeSpecName "kube-api-access-bcphd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:54.556746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.556711 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcphd\" (UniqueName: \"kubernetes.io/projected/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-kube-api-access-bcphd\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:54.556746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.556741 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb8fbc-4f0b-4288-a297-f8ce9d066613-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:54.588186 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:54.588099 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" path="/var/lib/kubelet/pods/1eaf8a64-6829-4034-89f8-a2608c2b7cc0/volumes" Nov 03 21:02:55.202474 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.202439 5815 generic.go:356] "Generic (PLEG): container finished" podID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerID="a3eb74e06640d93552b2cdf121b7259c7e0af267a25adbdce289d8c7359f00c0" exitCode=0 Nov 03 21:02:55.202671 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.202470 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" event={"ID":"0b300e92-11db-4ff9-b4fa-35fe3665d322","Type":"ContainerDied","Data":"a3eb74e06640d93552b2cdf121b7259c7e0af267a25adbdce289d8c7359f00c0"} Nov 03 21:02:55.205455 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.205419 5815 generic.go:356] "Generic (PLEG): container finished" podID="f7cb8fbc-4f0b-4288-a297-f8ce9d066613" containerID="8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293" exitCode=0 Nov 03 21:02:55.205653 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.205635 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" event={"ID":"f7cb8fbc-4f0b-4288-a297-f8ce9d066613","Type":"ContainerDied","Data":"8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293"} Nov 03 21:02:55.205747 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.205727 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" event={"ID":"f7cb8fbc-4f0b-4288-a297-f8ce9d066613","Type":"ContainerDied","Data":"bda263136d6d5b1ea5c7a7fb0919d7ad3b401cc0f24812d157f0fd73ad48a012"} Nov 03 21:02:55.205811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.205765 5815 scope.go:122] "RemoveContainer" containerID="8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293" Nov 03 21:02:55.205871 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.205864 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd" Nov 03 21:02:55.240936 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.240913 5815 scope.go:122] "RemoveContainer" containerID="8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293" Nov 03 21:02:55.241312 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:02:55.241293 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293\": container with ID starting with 8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293 not found: ID does not exist" containerID="8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293" Nov 03 21:02:55.241374 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.241318 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293"} err="failed to get container status \"8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293\": rpc error: code = NotFound desc = could not find container \"8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293\": container with ID starting with 8e7afc1104e2c605109f278e4b6dcb78a2d489bf7637f07ab821cb35f374f293 not found: ID does not exist" Nov 03 21:02:55.252412 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.252387 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd"] Nov 03 21:02:55.255900 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.255878 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bab2c3-5b846bbb44-nxgfd"] Nov 03 21:02:55.368878 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.368853 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:55.487804 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.487775 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.487821 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.487852 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.487905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.487947 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488113 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488160 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488216 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488222 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488244 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs\") pod \"0b300e92-11db-4ff9-b4fa-35fe3665d322\" (UID: \"0b300e92-11db-4ff9-b4fa-35fe3665d322\") " Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488442 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488745 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:55.488811 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488811 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.489853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488830 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b300e92-11db-4ff9-b4fa-35fe3665d322-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.489853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488845 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:55.489853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.488967 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:55.491078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.491024 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "kube-api-access-wjw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:55.491407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.491379 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:55.491407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.491398 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:55.491618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.491599 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo" pod "0b300e92-11db-4ff9-b4fa-35fe3665d322" (UID: "0b300e92-11db-4ff9-b4fa-35fe3665d322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:55.589608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589576 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjw97\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-kube-api-access-wjw97\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.589608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589600 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.589608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589609 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.590178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589618 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.590178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589626 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.590178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589636 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.590178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:55.589644 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b300e92-11db-4ff9-b4fa-35fe3665d322-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:56.215376 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.215345 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" Nov 03 21:02:56.215599 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.215344 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc" event={"ID":"0b300e92-11db-4ff9-b4fa-35fe3665d322","Type":"ContainerDied","Data":"79d5bdb11aa5ab24c3664f709729aee5e8a474d4f22093bfffab41b9e648ae99"} Nov 03 21:02:56.215599 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.215497 5815 scope.go:122] "RemoveContainer" containerID="a3eb74e06640d93552b2cdf121b7259c7e0af267a25adbdce289d8c7359f00c0" Nov 03 21:02:56.244702 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.244675 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc"] Nov 03 21:02:56.252181 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.252130 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-m2pnc"] Nov 03 21:02:56.585735 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.585663 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" path="/var/lib/kubelet/pods/0b300e92-11db-4ff9-b4fa-35fe3665d322/volumes" Nov 03 21:02:56.586119 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:56.586105 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7cb8fbc-4f0b-4288-a297-f8ce9d066613" path="/var/lib/kubelet/pods/f7cb8fbc-4f0b-4288-a297-f8ce9d066613/volumes" Nov 03 21:02:59.025493 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:59.025458 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc"] Nov 03 21:02:59.028416 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:02:59.028396 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b2b7183-8ptnc"] Nov 03 21:03:00.585329 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:00.585297 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db74f0de-2ad0-4c18-bd64-872e26ec682a" path="/var/lib/kubelet/pods/db74f0de-2ad0-4c18-bd64-872e26ec682a/volumes" Nov 03 21:03:07.084436 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:07.084395 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:03:13.852855 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:13.852817 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:03:21.374957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.374918 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8"] Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375443 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c226212b-ca3d-4b73-8ed5-719fe27c463d" containerName="payload-processing" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375454 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c226212b-ca3d-4b73-8ed5-719fe27c463d" containerName="payload-processing" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375463 5815 cpu_manager.go:414] "containerMap: removing container" podUID="f7cb8fbc-4f0b-4288-a297-f8ce9d066613" containerName="payload-processing" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375468 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cb8fbc-4f0b-4288-a297-f8ce9d066613" containerName="payload-processing" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375500 5815 cpu_manager.go:414] "containerMap: removing container" podUID="278413d0-ca91-4575-8120-a1514610eaf4" containerName="main" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375530 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278413d0-ca91-4575-8120-a1514610eaf4" containerName="main" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375542 5815 cpu_manager.go:414] "containerMap: removing container" podUID="70c47ff1-8f40-4082-bb80-b2752b747dfd" containerName="main" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375547 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c47ff1-8f40-4082-bb80-b2752b747dfd" containerName="main" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375555 5815 cpu_manager.go:414] "containerMap: removing container" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375560 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375570 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375574 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375667 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="278413d0-ca91-4575-8120-a1514610eaf4" containerName="main" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375678 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c47ff1-8f40-4082-bb80-b2752b747dfd" containerName="main" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375684 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eaf8a64-6829-4034-89f8-a2608c2b7cc0" containerName="istio-proxy" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375689 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b300e92-11db-4ff9-b4fa-35fe3665d322" containerName="istio-proxy" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375699 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c226212b-ca3d-4b73-8ed5-719fe27c463d" containerName="payload-processing" Nov 03 21:03:21.378880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.375704 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cb8fbc-4f0b-4288-a297-f8ce9d066613" containerName="payload-processing" Nov 03 21:03:21.395940 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.395906 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.400618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.400587 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:21.400751 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.400686 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:03:21.403038 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.402827 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8"] Nov 03 21:03:21.454642 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.454611 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.454767 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.454681 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjkpc\" (UniqueName: \"kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.454823 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.454770 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.555977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.555951 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkpc\" (UniqueName: \"kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.556158 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.556002 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.556158 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.556089 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.556685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.556664 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.558671 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.558646 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.568559 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.568492 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkpc\" (UniqueName: \"kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc\") pod \"maas-api-revoke-sub-864d592c-r7jd8\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:21.714590 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:21.714559 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:22.000319 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:22.000289 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8"] Nov 03 21:03:22.001023 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:03:22.000999 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578097e1_6482_4606_bcb4_8dd74dca55eb.slice/crio-1f29ca0e38508a2151a8129ed3a8af48ee1e4dcc9224cdb297e7cbc0769f86b2 WatchSource:0}: Error finding container 1f29ca0e38508a2151a8129ed3a8af48ee1e4dcc9224cdb297e7cbc0769f86b2: Status 404 returned error can't find the container with id 1f29ca0e38508a2151a8129ed3a8af48ee1e4dcc9224cdb297e7cbc0769f86b2 Nov 03 21:03:22.520215 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:22.520176 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" event={"ID":"578097e1-6482-4606-bcb4-8dd74dca55eb","Type":"ContainerStarted","Data":"48dd80457aeb832e9b0d939b106be4a7f7939fee3da961ac42496e67e24d92c5"} Nov 03 21:03:22.520657 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:22.520236 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" event={"ID":"578097e1-6482-4606-bcb4-8dd74dca55eb","Type":"ContainerStarted","Data":"1f29ca0e38508a2151a8129ed3a8af48ee1e4dcc9224cdb297e7cbc0769f86b2"} Nov 03 21:03:22.539948 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:22.539857 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" podStartSLOduration=1.539844159 podStartE2EDuration="1.539844159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:22.537816475 +0000 UTC m=+3712.068162145" watchObservedRunningTime="2026-11-03 21:03:22.539844159 +0000 UTC m=+3712.070189859" Nov 03 21:03:23.528733 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:23.528699 5815 generic.go:356] "Generic (PLEG): container finished" podID="578097e1-6482-4606-bcb4-8dd74dca55eb" containerID="48dd80457aeb832e9b0d939b106be4a7f7939fee3da961ac42496e67e24d92c5" exitCode=0 Nov 03 21:03:23.529218 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:23.528780 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" event={"ID":"578097e1-6482-4606-bcb4-8dd74dca55eb","Type":"ContainerDied","Data":"48dd80457aeb832e9b0d939b106be4a7f7939fee3da961ac42496e67e24d92c5"} Nov 03 21:03:24.441681 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.441651 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:03:24.683779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.683749 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:24.790576 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.790537 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca\") pod \"578097e1-6482-4606-bcb4-8dd74dca55eb\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " Nov 03 21:03:24.790803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.790712 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc\" (UniqueName: \"kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc\") pod \"578097e1-6482-4606-bcb4-8dd74dca55eb\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " Nov 03 21:03:24.790884 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.790803 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca\") pod \"578097e1-6482-4606-bcb4-8dd74dca55eb\" (UID: \"578097e1-6482-4606-bcb4-8dd74dca55eb\") " Nov 03 21:03:24.791201 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.791172 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca" pod "578097e1-6482-4606-bcb4-8dd74dca55eb" (UID: "578097e1-6482-4606-bcb4-8dd74dca55eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:24.791538 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.791487 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:24.793085 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.793047 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc" pod "578097e1-6482-4606-bcb4-8dd74dca55eb" (UID: "578097e1-6482-4606-bcb4-8dd74dca55eb"). InnerVolumeSpecName "kube-api-access-vjkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:24.793883 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.793862 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca" pod "578097e1-6482-4606-bcb4-8dd74dca55eb" (UID: "578097e1-6482-4606-bcb4-8dd74dca55eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:24.892598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.892566 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjkpc\" (UniqueName: \"kubernetes.io/projected/578097e1-6482-4606-bcb4-8dd74dca55eb-kube-api-access-vjkpc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:24.892598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:24.892594 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578097e1-6482-4606-bcb4-8dd74dca55eb-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.547409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:25.547379 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" Nov 03 21:03:25.547677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:25.547383 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8" event={"ID":"578097e1-6482-4606-bcb4-8dd74dca55eb","Type":"ContainerDied","Data":"1f29ca0e38508a2151a8129ed3a8af48ee1e4dcc9224cdb297e7cbc0769f86b2"} Nov 03 21:03:25.547677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:25.547492 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f29ca0e38508a2151a8129ed3a8af48ee1e4dcc9224cdb297e7cbc0769f86b2" Nov 03 21:03:25.933304 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:25.933218 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2"] Nov 03 21:03:25.951233 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:25.951202 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2"] Nov 03 21:03:28.490350 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.490317 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c"] Nov 03 21:03:28.490916 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.490900 5815 cpu_manager.go:414] "containerMap: removing container" podUID="578097e1-6482-4606-bcb4-8dd74dca55eb" containerName="revoke-keys" Nov 03 21:03:28.490916 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.490917 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578097e1-6482-4606-bcb4-8dd74dca55eb" containerName="revoke-keys" Nov 03 21:03:28.491036 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.491008 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="578097e1-6482-4606-bcb4-8dd74dca55eb" containerName="revoke-keys" Nov 03 21:03:28.494699 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.494678 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.502432 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.500723 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:03:28.502432 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.500794 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:28.513323 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.505748 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c"] Nov 03 21:03:28.672288 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.672197 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.672461 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.672347 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.672461 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.672415 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.773448 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.773393 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.773657 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.773492 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.773740 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.773669 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.774595 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.774569 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.776743 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.776711 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.786054 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.786028 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr\") pod \"maas-api-revoke-sub-696db94b-6hb9c\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:28.815038 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:28.815015 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:29.101343 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:03:29.101312 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73336eb9_eaea_48b1_b58e_361c5d908aa1.slice/crio-8e138dbe1ae23c6843d1835ad847060d2276e5747a647a1da6f72f92e9374190 WatchSource:0}: Error finding container 8e138dbe1ae23c6843d1835ad847060d2276e5747a647a1da6f72f92e9374190: Status 404 returned error can't find the container with id 8e138dbe1ae23c6843d1835ad847060d2276e5747a647a1da6f72f92e9374190 Nov 03 21:03:29.101621 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:29.101594 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c"] Nov 03 21:03:29.188364 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:29.188322 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47"] Nov 03 21:03:29.203039 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:29.203007 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6"] Nov 03 21:03:29.579689 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:29.579644 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" event={"ID":"73336eb9-eaea-48b1-b58e-361c5d908aa1","Type":"ContainerStarted","Data":"5004bb90068d47dd7f75ed9231cdf7aaf2868047c866a5f69cc10deed152cc23"} Nov 03 21:03:29.580168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:29.579744 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" event={"ID":"73336eb9-eaea-48b1-b58e-361c5d908aa1","Type":"ContainerStarted","Data":"8e138dbe1ae23c6843d1835ad847060d2276e5747a647a1da6f72f92e9374190"} Nov 03 21:03:29.599181 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:29.599130 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" podStartSLOduration=1.599116493 podStartE2EDuration="1.599116493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:29.598069169 +0000 UTC m=+3719.128414845" watchObservedRunningTime="2026-11-03 21:03:29.599116493 +0000 UTC m=+3719.129462165" Nov 03 21:03:30.601072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:30.601032 5815 generic.go:356] "Generic (PLEG): container finished" podID="73336eb9-eaea-48b1-b58e-361c5d908aa1" containerID="5004bb90068d47dd7f75ed9231cdf7aaf2868047c866a5f69cc10deed152cc23" exitCode=0 Nov 03 21:03:30.602030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:30.601996 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" event={"ID":"73336eb9-eaea-48b1-b58e-361c5d908aa1","Type":"ContainerDied","Data":"5004bb90068d47dd7f75ed9231cdf7aaf2868047c866a5f69cc10deed152cc23"} Nov 03 21:03:31.786163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.786135 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:31.810432 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.810402 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr\") pod \"73336eb9-eaea-48b1-b58e-361c5d908aa1\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " Nov 03 21:03:31.810758 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.810736 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca\") pod \"73336eb9-eaea-48b1-b58e-361c5d908aa1\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " Nov 03 21:03:31.810986 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.810924 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca\") pod \"73336eb9-eaea-48b1-b58e-361c5d908aa1\" (UID: \"73336eb9-eaea-48b1-b58e-361c5d908aa1\") " Nov 03 21:03:31.811121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.811075 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca" pod "73336eb9-eaea-48b1-b58e-361c5d908aa1" (UID: "73336eb9-eaea-48b1-b58e-361c5d908aa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:31.811401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.811377 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:31.813427 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.813400 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr" pod "73336eb9-eaea-48b1-b58e-361c5d908aa1" (UID: "73336eb9-eaea-48b1-b58e-361c5d908aa1"). InnerVolumeSpecName "kube-api-access-h5qmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:31.815779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.813746 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca" pod "73336eb9-eaea-48b1-b58e-361c5d908aa1" (UID: "73336eb9-eaea-48b1-b58e-361c5d908aa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:31.912296 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.912236 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73336eb9-eaea-48b1-b58e-361c5d908aa1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:31.912296 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:31.912261 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5qmr\" (UniqueName: \"kubernetes.io/projected/73336eb9-eaea-48b1-b58e-361c5d908aa1-kube-api-access-h5qmr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:32.111665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.111619 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv"] Nov 03 21:03:32.112195 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.112178 5815 cpu_manager.go:414] "containerMap: removing container" podUID="73336eb9-eaea-48b1-b58e-361c5d908aa1" containerName="revoke-keys" Nov 03 21:03:32.112289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.112198 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73336eb9-eaea-48b1-b58e-361c5d908aa1" containerName="revoke-keys" Nov 03 21:03:32.112361 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.112323 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="73336eb9-eaea-48b1-b58e-361c5d908aa1" containerName="revoke-keys" Nov 03 21:03:32.115932 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.115911 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.123973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.123943 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv"] Nov 03 21:03:32.215977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.215945 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgnkc\" (UniqueName: \"kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.215977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.215980 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.216249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.216021 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.331833 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.326972 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgnkc\" (UniqueName: \"kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.331833 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.327031 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.331833 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.327125 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.331833 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.331493 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.347989 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.347104 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.352204 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.352174 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgnkc\" (UniqueName: \"kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc\") pod \"maas-api-revoke-sub-b05cbe9f-7wbqv\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.436066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.436037 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:32.636167 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.636140 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" Nov 03 21:03:32.636342 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.636174 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c" event={"ID":"73336eb9-eaea-48b1-b58e-361c5d908aa1","Type":"ContainerDied","Data":"8e138dbe1ae23c6843d1835ad847060d2276e5747a647a1da6f72f92e9374190"} Nov 03 21:03:32.636342 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.636221 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e138dbe1ae23c6843d1835ad847060d2276e5747a647a1da6f72f92e9374190" Nov 03 21:03:32.719491 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:32.719466 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv"] Nov 03 21:03:32.720880 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:03:32.720849 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eccc426_4b20_485d_8a90_f6e7174f4788.slice/crio-3c08ac620dd1d0fdb34da36521ef0ed88e0e0fa1088422b378d155f81ff693e6 WatchSource:0}: Error finding container 3c08ac620dd1d0fdb34da36521ef0ed88e0e0fa1088422b378d155f81ff693e6: Status 404 returned error can't find the container with id 3c08ac620dd1d0fdb34da36521ef0ed88e0e0fa1088422b378d155f81ff693e6 Nov 03 21:03:33.646350 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:33.646316 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" event={"ID":"5eccc426-4b20-485d-8a90-f6e7174f4788","Type":"ContainerStarted","Data":"d8d5aade4f7649d59f6a677d75bbc7081421eea8596ba6dd989a1eca75999fb4"} Nov 03 21:03:33.646882 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:33.646373 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" event={"ID":"5eccc426-4b20-485d-8a90-f6e7174f4788","Type":"ContainerStarted","Data":"3c08ac620dd1d0fdb34da36521ef0ed88e0e0fa1088422b378d155f81ff693e6"} Nov 03 21:03:33.670614 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:33.670559 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" podStartSLOduration=1.6705408830000001 podStartE2EDuration="1.670540883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:33.667413524 +0000 UTC m=+3723.197759196" watchObservedRunningTime="2026-11-03 21:03:33.670540883 +0000 UTC m=+3723.200886557" Nov 03 21:03:34.204199 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:34.204163 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerName="payload-pre-processing" containerID="cri-o://2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122" gracePeriod=25 Nov 03 21:03:34.579794 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:34.579757 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerName="payload-processing" containerID="cri-o://631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21" gracePeriod=25 Nov 03 21:03:34.663913 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:34.663873 5815 generic.go:356] "Generic (PLEG): container finished" podID="5eccc426-4b20-485d-8a90-f6e7174f4788" containerID="d8d5aade4f7649d59f6a677d75bbc7081421eea8596ba6dd989a1eca75999fb4" exitCode=0 Nov 03 21:03:34.664495 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:34.663968 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" event={"ID":"5eccc426-4b20-485d-8a90-f6e7174f4788","Type":"ContainerDied","Data":"d8d5aade4f7649d59f6a677d75bbc7081421eea8596ba6dd989a1eca75999fb4"} Nov 03 21:03:35.845121 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.845091 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:35.892401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.892360 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca\") pod \"5eccc426-4b20-485d-8a90-f6e7174f4788\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " Nov 03 21:03:35.894108 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.894032 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca" pod "5eccc426-4b20-485d-8a90-f6e7174f4788" (UID: "5eccc426-4b20-485d-8a90-f6e7174f4788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:35.894518 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.894473 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca\") pod \"5eccc426-4b20-485d-8a90-f6e7174f4788\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " Nov 03 21:03:35.894826 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.894765 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc\" (UniqueName: \"kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc\") pod \"5eccc426-4b20-485d-8a90-f6e7174f4788\" (UID: \"5eccc426-4b20-485d-8a90-f6e7174f4788\") " Nov 03 21:03:35.895220 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.895196 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:35.897131 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.897093 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca" pod "5eccc426-4b20-485d-8a90-f6e7174f4788" (UID: "5eccc426-4b20-485d-8a90-f6e7174f4788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:35.897593 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.897567 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc" pod "5eccc426-4b20-485d-8a90-f6e7174f4788" (UID: "5eccc426-4b20-485d-8a90-f6e7174f4788"). InnerVolumeSpecName "kube-api-access-fgnkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:35.995923 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.995898 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eccc426-4b20-485d-8a90-f6e7174f4788-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:35.995923 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:35.995923 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgnkc\" (UniqueName: \"kubernetes.io/projected/5eccc426-4b20-485d-8a90-f6e7174f4788-kube-api-access-fgnkc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:36.085349 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:36.085316 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:36.086110 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:36.086086 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:36.690187 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:36.690158 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" Nov 03 21:03:36.690407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:36.690201 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv" event={"ID":"5eccc426-4b20-485d-8a90-f6e7174f4788","Type":"ContainerDied","Data":"3c08ac620dd1d0fdb34da36521ef0ed88e0e0fa1088422b378d155f81ff693e6"} Nov 03 21:03:36.690407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:36.690256 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c08ac620dd1d0fdb34da36521ef0ed88e0e0fa1088422b378d155f81ff693e6" Nov 03 21:03:38.697228 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.697194 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs"] Nov 03 21:03:38.698034 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.698016 5815 cpu_manager.go:414] "containerMap: removing container" podUID="5eccc426-4b20-485d-8a90-f6e7174f4788" containerName="revoke-keys" Nov 03 21:03:38.698090 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.698037 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eccc426-4b20-485d-8a90-f6e7174f4788" containerName="revoke-keys" Nov 03 21:03:38.698198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.698186 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eccc426-4b20-485d-8a90-f6e7174f4788" containerName="revoke-keys" Nov 03 21:03:38.702534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.702483 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.710812 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.710786 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs"] Nov 03 21:03:38.712611 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.712582 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:03:38.712860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.712841 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:38.840647 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.840608 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ps9\" (UniqueName: \"kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.840647 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.840650 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.840888 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.840740 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.941924 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.941889 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ps9\" (UniqueName: \"kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.942082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.941932 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.942082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.941973 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.942643 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.942621 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.944662 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.944640 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:38.953874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:38.953813 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ps9\" (UniqueName: \"kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9\") pod \"maas-api-revoke-sub-7b457452-qp9xs\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:39.028029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:39.028006 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:39.320918 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:39.320886 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs"] Nov 03 21:03:39.321122 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:03:39.321101 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded13b7f1_a0d2_4ff8_b130_72069b89b501.slice/crio-f5cf52a8a1f31870c6f657ef509b7801aec2cae45b87c3b4d622c94b604d8fcc WatchSource:0}: Error finding container f5cf52a8a1f31870c6f657ef509b7801aec2cae45b87c3b4d622c94b604d8fcc: Status 404 returned error can't find the container with id f5cf52a8a1f31870c6f657ef509b7801aec2cae45b87c3b4d622c94b604d8fcc Nov 03 21:03:39.732133 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:39.732102 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" event={"ID":"ed13b7f1-a0d2-4ff8-b130-72069b89b501","Type":"ContainerStarted","Data":"18601fea8e5c80a308a14f6c56f4308ac9a63b045891ad0a99ce6d0d33ca1594"} Nov 03 21:03:39.732556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:39.732152 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" event={"ID":"ed13b7f1-a0d2-4ff8-b130-72069b89b501","Type":"ContainerStarted","Data":"f5cf52a8a1f31870c6f657ef509b7801aec2cae45b87c3b4d622c94b604d8fcc"} Nov 03 21:03:39.750573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:39.750484 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" podStartSLOduration=1.7504676510000001 podStartE2EDuration="1.750467651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:39.74933681 +0000 UTC m=+3729.279682483" watchObservedRunningTime="2026-11-03 21:03:39.750467651 +0000 UTC m=+3729.280813322" Nov 03 21:03:40.741334 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:40.741296 5815 generic.go:356] "Generic (PLEG): container finished" podID="ed13b7f1-a0d2-4ff8-b130-72069b89b501" containerID="18601fea8e5c80a308a14f6c56f4308ac9a63b045891ad0a99ce6d0d33ca1594" exitCode=0 Nov 03 21:03:40.741804 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:40.741381 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" event={"ID":"ed13b7f1-a0d2-4ff8-b130-72069b89b501","Type":"ContainerDied","Data":"18601fea8e5c80a308a14f6c56f4308ac9a63b045891ad0a99ce6d0d33ca1594"} Nov 03 21:03:40.960367 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:40.960329 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" containerID="cri-o://8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17" gracePeriod=45 Nov 03 21:03:40.983176 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:40.983131 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" containerID="cri-o://b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e" gracePeriod=45 Nov 03 21:03:41.255556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.255526 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 21:03:41.271549 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.271471 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 21:03:41.370648 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370617 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache\") pod \"3870873f-d57a-4a3b-b82a-25d596f20c80\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " Nov 03 21:03:41.370853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370684 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm\") pod \"3870873f-d57a-4a3b-b82a-25d596f20c80\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " Nov 03 21:03:41.370853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370764 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache\") pod \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " Nov 03 21:03:41.370853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370820 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs\" (UniqueName: \"kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs\") pod \"3870873f-d57a-4a3b-b82a-25d596f20c80\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " Nov 03 21:03:41.370853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370851 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm\") pod \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " Nov 03 21:03:41.371095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370935 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs\") pod \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " Nov 03 21:03:41.371095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.370932 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache" pod "3870873f-d57a-4a3b-b82a-25d596f20c80" (UID: "3870873f-d57a-4a3b-b82a-25d596f20c80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.371219 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371161 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache" pod "6e15d366-3ba2-487e-8c5e-0d0cd4b65040" (UID: "6e15d366-3ba2-487e-8c5e-0d0cd4b65040"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.371363 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371341 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home\") pod \"3870873f-d57a-4a3b-b82a-25d596f20c80\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " Nov 03 21:03:41.371460 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371395 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir\") pod \"3870873f-d57a-4a3b-b82a-25d596f20c80\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " Nov 03 21:03:41.371460 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371435 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h\" (UniqueName: \"kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h\") pod \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " Nov 03 21:03:41.371612 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371496 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir\") pod \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " Nov 03 21:03:41.371612 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371564 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home\") pod \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\" (UID: \"6e15d366-3ba2-487e-8c5e-0d0cd4b65040\") " Nov 03 21:03:41.371612 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371595 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s\" (UniqueName: \"kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s\") pod \"3870873f-d57a-4a3b-b82a-25d596f20c80\" (UID: \"3870873f-d57a-4a3b-b82a-25d596f20c80\") " Nov 03 21:03:41.371965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.371800 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir" pod "6e15d366-3ba2-487e-8c5e-0d0cd4b65040" (UID: "6e15d366-3ba2-487e-8c5e-0d0cd4b65040"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.372101 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.372077 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir" pod "3870873f-d57a-4a3b-b82a-25d596f20c80" (UID: "3870873f-d57a-4a3b-b82a-25d596f20c80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.372226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.372186 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.372226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.372213 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.372418 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.372229 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.372418 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.372278 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home" pod "6e15d366-3ba2-487e-8c5e-0d0cd4b65040" (UID: "6e15d366-3ba2-487e-8c5e-0d0cd4b65040"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.372643 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.372453 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home" pod "3870873f-d57a-4a3b-b82a-25d596f20c80" (UID: "3870873f-d57a-4a3b-b82a-25d596f20c80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.374031 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.374002 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs" pod "6e15d366-3ba2-487e-8c5e-0d0cd4b65040" (UID: "6e15d366-3ba2-487e-8c5e-0d0cd4b65040"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:41.374902 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.374860 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs" pod "3870873f-d57a-4a3b-b82a-25d596f20c80" (UID: "3870873f-d57a-4a3b-b82a-25d596f20c80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:41.375017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.374904 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm" pod "6e15d366-3ba2-487e-8c5e-0d0cd4b65040" (UID: "6e15d366-3ba2-487e-8c5e-0d0cd4b65040"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.375242 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.375222 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm" pod "3870873f-d57a-4a3b-b82a-25d596f20c80" (UID: "3870873f-d57a-4a3b-b82a-25d596f20c80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.375462 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.375441 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s" pod "3870873f-d57a-4a3b-b82a-25d596f20c80" (UID: "3870873f-d57a-4a3b-b82a-25d596f20c80"). InnerVolumeSpecName "kube-api-access-bxd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:41.376343 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.376318 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h" pod "6e15d366-3ba2-487e-8c5e-0d0cd4b65040" (UID: "6e15d366-3ba2-487e-8c5e-0d0cd4b65040"). InnerVolumeSpecName "kube-api-access-x662h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:41.473971 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.473929 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.473971 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.473969 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3870873f-d57a-4a3b-b82a-25d596f20c80-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.473982 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.473994 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.474010 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.474020 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3870873f-d57a-4a3b-b82a-25d596f20c80-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.474035 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x662h\" (UniqueName: \"kubernetes.io/projected/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-kube-api-access-x662h\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.474049 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e15d366-3ba2-487e-8c5e-0d0cd4b65040-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.474198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.474061 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxd4s\" (UniqueName: \"kubernetes.io/projected/3870873f-d57a-4a3b-b82a-25d596f20c80-kube-api-access-bxd4s\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.557255 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.557179 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:03:41.749990 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.749954 5815 generic.go:356] "Generic (PLEG): container finished" podID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerID="8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17" exitCode=0 Nov 03 21:03:41.750488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.749999 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" Nov 03 21:03:41.750488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.750030 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" event={"ID":"6e15d366-3ba2-487e-8c5e-0d0cd4b65040","Type":"ContainerDied","Data":"8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17"} Nov 03 21:03:41.750488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.750076 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" event={"ID":"6e15d366-3ba2-487e-8c5e-0d0cd4b65040","Type":"ContainerDied","Data":"50c36e3b16efc6e11004a32c241039722f9842eb31e6e2986031fb629d94cefe"} Nov 03 21:03:41.750488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.750093 5815 scope.go:122] "RemoveContainer" containerID="8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17" Nov 03 21:03:41.751830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.751801 5815 generic.go:356] "Generic (PLEG): container finished" podID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerID="b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e" exitCode=0 Nov 03 21:03:41.751969 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.751849 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" Nov 03 21:03:41.751969 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.751887 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" event={"ID":"3870873f-d57a-4a3b-b82a-25d596f20c80","Type":"ContainerDied","Data":"b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e"} Nov 03 21:03:41.751969 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.751923 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2" event={"ID":"3870873f-d57a-4a3b-b82a-25d596f20c80","Type":"ContainerDied","Data":"d8ffcddfa8f4b1ed5a27051b2b62acd443f5f73d9a7033996df6ef2d61629f22"} Nov 03 21:03:41.762163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.762142 5815 scope.go:122] "RemoveContainer" containerID="8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17" Nov 03 21:03:41.762494 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:41.762468 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17\": container with ID starting with 8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17 not found: ID does not exist" containerID="8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17" Nov 03 21:03:41.762674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.762549 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17"} err="failed to get container status \"8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17\": rpc error: code = NotFound desc = could not find container \"8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17\": container with ID starting with 8fab78670f1591cfbea51ec7e27daa84088d064e8ccd38cc0b18aea456c65b17 not found: ID does not exist" Nov 03 21:03:41.762674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.762574 5815 scope.go:122] "RemoveContainer" containerID="b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e" Nov 03 21:03:41.775115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.775098 5815 scope.go:122] "RemoveContainer" containerID="b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e" Nov 03 21:03:41.775710 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:41.775420 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e\": container with ID starting with b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e not found: ID does not exist" containerID="b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e" Nov 03 21:03:41.775710 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.775456 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e"} err="failed to get container status \"b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e\": rpc error: code = NotFound desc = could not find container \"b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e\": container with ID starting with b010acc4bccf16cd0db9d1827e28c93c113763918d8ef2a844defe75fa6c0b4e not found: ID does not exist" Nov 03 21:03:41.779782 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.779761 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2"] Nov 03 21:03:41.789063 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.789033 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/simulator-w3-4f93d1-kserve-79468dcbcc-dmqk2"] Nov 03 21:03:41.799783 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.799763 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2"] Nov 03 21:03:41.806534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.806498 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2"] Nov 03 21:03:41.890932 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.890903 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:41.980501 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.980477 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9\" (UniqueName: \"kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9\") pod \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " Nov 03 21:03:41.980681 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.980576 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca\") pod \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " Nov 03 21:03:41.980755 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.980692 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca\") pod \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\" (UID: \"ed13b7f1-a0d2-4ff8-b130-72069b89b501\") " Nov 03 21:03:41.981141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.981100 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca" pod "ed13b7f1-a0d2-4ff8-b130-72069b89b501" (UID: "ed13b7f1-a0d2-4ff8-b130-72069b89b501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:41.983089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.983063 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca" pod "ed13b7f1-a0d2-4ff8-b130-72069b89b501" (UID: "ed13b7f1-a0d2-4ff8-b130-72069b89b501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:41.983199 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:41.983100 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9" pod "ed13b7f1-a0d2-4ff8-b130-72069b89b501" (UID: "ed13b7f1-a0d2-4ff8-b130-72069b89b501"). InnerVolumeSpecName "kube-api-access-s2ps9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:42.082711 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.082633 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.082711 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.082660 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ps9\" (UniqueName: \"kubernetes.io/projected/ed13b7f1-a0d2-4ff8-b130-72069b89b501-kube-api-access-s2ps9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.082711 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.082670 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed13b7f1-a0d2-4ff8-b130-72069b89b501-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.153164 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.153123 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-w3-4f93d1-kserve-945b8987-7qhs2" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:42.628791 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.628753 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" path="/var/lib/kubelet/pods/3870873f-d57a-4a3b-b82a-25d596f20c80/volumes" Nov 03 21:03:42.629209 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.629195 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" path="/var/lib/kubelet/pods/6e15d366-3ba2-487e-8c5e-0d0cd4b65040/volumes" Nov 03 21:03:42.761012 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.760984 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" Nov 03 21:03:42.761626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.760986 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs" event={"ID":"ed13b7f1-a0d2-4ff8-b130-72069b89b501","Type":"ContainerDied","Data":"f5cf52a8a1f31870c6f657ef509b7801aec2cae45b87c3b4d622c94b604d8fcc"} Nov 03 21:03:42.761626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:42.761143 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5cf52a8a1f31870c6f657ef509b7801aec2cae45b87c3b4d622c94b604d8fcc" Nov 03 21:03:43.852582 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:43.852541 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:03:44.768634 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:44.768599 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7"] Nov 03 21:03:44.798855 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:44.797887 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq"] Nov 03 21:03:46.086021 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:46.085982 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:46.086430 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:46.085996 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:47.056362 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:47.056329 5815 scope.go:122] "RemoveContainer" containerID="323139ee6cae8876b130c027a0f4f7bc6d3b96cb2ea3b30b12861525fc289d42" Nov 03 21:03:49.358240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.358197 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6"] Nov 03 21:03:49.358956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.358932 5815 cpu_manager.go:414] "containerMap: removing container" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" Nov 03 21:03:49.359066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.358958 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" Nov 03 21:03:49.359066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359004 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" Nov 03 21:03:49.359066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359012 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" Nov 03 21:03:49.359066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359052 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ed13b7f1-a0d2-4ff8-b130-72069b89b501" containerName="revoke-keys" Nov 03 21:03:49.359066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359061 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed13b7f1-a0d2-4ff8-b130-72069b89b501" containerName="revoke-keys" Nov 03 21:03:49.359352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359216 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e15d366-3ba2-487e-8c5e-0d0cd4b65040" containerName="main" Nov 03 21:03:49.359352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359235 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3870873f-d57a-4a3b-b82a-25d596f20c80" containerName="main" Nov 03 21:03:49.359352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.359247 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed13b7f1-a0d2-4ff8-b130-72069b89b501" containerName="revoke-keys" Nov 03 21:03:49.364159 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.364137 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.369189 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.369163 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:49.369398 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.369374 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:03:49.371955 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.371932 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6"] Nov 03 21:03:49.468945 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.468915 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.469124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.469071 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.469176 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.469149 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fmq\" (UniqueName: \"kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.579144 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.579101 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.579375 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.579184 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fmq\" (UniqueName: \"kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.579375 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.579283 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.580475 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.580445 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.582920 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.582866 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.611580 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.598583 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fmq\" (UniqueName: \"kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq\") pod \"maas-api-revoke-sub-839954d0-gsdf6\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.681217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.681174 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:49.770058 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.769879 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" podUID="e3057a3a-a231-4340-8752-d6d7b670f007" containerName="payload-processing" containerID="cri-o://ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5" gracePeriod=25 Nov 03 21:03:49.799602 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.799545 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" containerName="payload-pre-processing" containerID="cri-o://54987fae2140f60b231ae3ec200c1cf00382f0672104e18342468afc91e65023" gracePeriod=25 Nov 03 21:03:49.975041 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:49.975008 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6"] Nov 03 21:03:49.979218 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:03:49.979179 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0db79b_4963_47ec_897b_d3248f62d373.slice/crio-74a641abe3d8a02b4c3e10d9ed0f5d193997cef0f9cffb434d919dc1873bd520 WatchSource:0}: Error finding container 74a641abe3d8a02b4c3e10d9ed0f5d193997cef0f9cffb434d919dc1873bd520: Status 404 returned error can't find the container with id 74a641abe3d8a02b4c3e10d9ed0f5d193997cef0f9cffb434d919dc1873bd520 Nov 03 21:03:50.025554 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.025532 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 21:03:50.187642 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.187616 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f\" (UniqueName: \"kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f\") pod \"e3057a3a-a231-4340-8752-d6d7b670f007\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " Nov 03 21:03:50.187768 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.187690 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume\") pod \"e3057a3a-a231-4340-8752-d6d7b670f007\" (UID: \"e3057a3a-a231-4340-8752-d6d7b670f007\") " Nov 03 21:03:50.188130 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.188105 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume" pod "e3057a3a-a231-4340-8752-d6d7b670f007" (UID: "e3057a3a-a231-4340-8752-d6d7b670f007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:50.190069 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.190043 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f" pod "e3057a3a-a231-4340-8752-d6d7b670f007" (UID: "e3057a3a-a231-4340-8752-d6d7b670f007"). InnerVolumeSpecName "kube-api-access-xcw7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:50.309602 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.309379 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3057a3a-a231-4340-8752-d6d7b670f007-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.309602 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.309418 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcw7f\" (UniqueName: \"kubernetes.io/projected/e3057a3a-a231-4340-8752-d6d7b670f007-kube-api-access-xcw7f\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.859494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.859454 5815 generic.go:356] "Generic (PLEG): container finished" podID="e3057a3a-a231-4340-8752-d6d7b670f007" containerID="ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5" exitCode=0 Nov 03 21:03:50.860003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.859564 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" Nov 03 21:03:50.860003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.859564 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" event={"ID":"e3057a3a-a231-4340-8752-d6d7b670f007","Type":"ContainerDied","Data":"ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5"} Nov 03 21:03:50.860003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.859683 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7" event={"ID":"e3057a3a-a231-4340-8752-d6d7b670f007","Type":"ContainerDied","Data":"b67587e53d43545c0f7eaa91601ea455519791a026abef17e65f6c7947eee97f"} Nov 03 21:03:50.860003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.859707 5815 scope.go:122] "RemoveContainer" containerID="ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5" Nov 03 21:03:50.861877 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.861851 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" event={"ID":"3c0db79b-4963-47ec-897b-d3248f62d373","Type":"ContainerStarted","Data":"c03a982609098ab2b1e99fd368f7aa41d9ef3696b75dd4533513e5aaa58f8c71"} Nov 03 21:03:50.862052 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.861937 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" event={"ID":"3c0db79b-4963-47ec-897b-d3248f62d373","Type":"ContainerStarted","Data":"74a641abe3d8a02b4c3e10d9ed0f5d193997cef0f9cffb434d919dc1873bd520"} Nov 03 21:03:50.871455 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.871436 5815 scope.go:122] "RemoveContainer" containerID="ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5" Nov 03 21:03:50.871793 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:50.871773 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5\": container with ID starting with ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5 not found: ID does not exist" containerID="ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5" Nov 03 21:03:50.871870 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.871801 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5"} err="failed to get container status \"ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5\": rpc error: code = NotFound desc = could not find container \"ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5\": container with ID starting with ecf073c86c29a192b07e338121d1d3a5ced2936acd0d7ace98fcbf6d90201ae5 not found: ID does not exist" Nov 03 21:03:50.888157 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.887941 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" podStartSLOduration=1.8879270689999998 podStartE2EDuration="1.887927069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:50.883170854 +0000 UTC m=+3740.413516527" watchObservedRunningTime="2026-11-03 21:03:50.887927069 +0000 UTC m=+3740.418272731" Nov 03 21:03:50.899445 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.899414 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7"] Nov 03 21:03:50.902962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:50.902943 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-37aa95-68669fc94d-867j7"] Nov 03 21:03:51.871441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:51.871398 5815 generic.go:356] "Generic (PLEG): container finished" podID="3c0db79b-4963-47ec-897b-d3248f62d373" containerID="c03a982609098ab2b1e99fd368f7aa41d9ef3696b75dd4533513e5aaa58f8c71" exitCode=0 Nov 03 21:03:51.871918 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:51.871485 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" event={"ID":"3c0db79b-4963-47ec-897b-d3248f62d373","Type":"ContainerDied","Data":"c03a982609098ab2b1e99fd368f7aa41d9ef3696b75dd4533513e5aaa58f8c71"} Nov 03 21:03:52.585806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:52.585770 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3057a3a-a231-4340-8752-d6d7b670f007" path="/var/lib/kubelet/pods/e3057a3a-a231-4340-8752-d6d7b670f007/volumes" Nov 03 21:03:53.046921 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.046895 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:53.052614 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.052586 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca\") pod \"3c0db79b-4963-47ec-897b-d3248f62d373\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " Nov 03 21:03:53.052784 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.052711 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca\") pod \"3c0db79b-4963-47ec-897b-d3248f62d373\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " Nov 03 21:03:53.052904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.052822 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq\" (UniqueName: \"kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq\") pod \"3c0db79b-4963-47ec-897b-d3248f62d373\" (UID: \"3c0db79b-4963-47ec-897b-d3248f62d373\") " Nov 03 21:03:53.053033 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.053012 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca" pod "3c0db79b-4963-47ec-897b-d3248f62d373" (UID: "3c0db79b-4963-47ec-897b-d3248f62d373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:53.053298 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.053279 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:53.055545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.055491 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca" pod "3c0db79b-4963-47ec-897b-d3248f62d373" (UID: "3c0db79b-4963-47ec-897b-d3248f62d373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:53.055726 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.055707 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq" pod "3c0db79b-4963-47ec-897b-d3248f62d373" (UID: "3c0db79b-4963-47ec-897b-d3248f62d373"). InnerVolumeSpecName "kube-api-access-t7fmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:53.154256 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.154181 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c0db79b-4963-47ec-897b-d3248f62d373-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:53.154256 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.154213 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7fmq\" (UniqueName: \"kubernetes.io/projected/3c0db79b-4963-47ec-897b-d3248f62d373-kube-api-access-t7fmq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:53.277673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.277636 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh"] Nov 03 21:03:53.277928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.277903 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" containerID="cri-o://ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53" gracePeriod=30 Nov 03 21:03:53.888559 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.888526 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" Nov 03 21:03:53.888774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.888558 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6" event={"ID":"3c0db79b-4963-47ec-897b-d3248f62d373","Type":"ContainerDied","Data":"74a641abe3d8a02b4c3e10d9ed0f5d193997cef0f9cffb434d919dc1873bd520"} Nov 03 21:03:53.888774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:53.888608 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74a641abe3d8a02b4c3e10d9ed0f5d193997cef0f9cffb434d919dc1873bd520" Nov 03 21:03:54.499584 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.499560 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 21:03:54.584954 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.584921 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume\") pod \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " Nov 03 21:03:54.585141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.585114 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb\" (UniqueName: \"kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb\") pod \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\" (UID: \"c693e7b8-e44b-4fd8-843e-0a686c5fcce2\") " Nov 03 21:03:54.585343 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.585318 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume" pod "c693e7b8-e44b-4fd8-843e-0a686c5fcce2" (UID: "c693e7b8-e44b-4fd8-843e-0a686c5fcce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:54.585633 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.585611 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:54.587696 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.587633 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb" pod "c693e7b8-e44b-4fd8-843e-0a686c5fcce2" (UID: "c693e7b8-e44b-4fd8-843e-0a686c5fcce2"). InnerVolumeSpecName "kube-api-access-jtrnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:54.687263 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.687237 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrnb\" (UniqueName: \"kubernetes.io/projected/c693e7b8-e44b-4fd8-843e-0a686c5fcce2-kube-api-access-jtrnb\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:54.843444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.843420 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 21:03:54.888985 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.888951 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q\" (UniqueName: \"kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q\") pod \"572f9e0c-4664-4a36-a52f-7936058aa1e4\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " Nov 03 21:03:54.889168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.889054 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume\") pod \"572f9e0c-4664-4a36-a52f-7936058aa1e4\" (UID: \"572f9e0c-4664-4a36-a52f-7936058aa1e4\") " Nov 03 21:03:54.889585 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.889554 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume" pod "572f9e0c-4664-4a36-a52f-7936058aa1e4" (UID: "572f9e0c-4664-4a36-a52f-7936058aa1e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:54.892035 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.892005 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q" pod "572f9e0c-4664-4a36-a52f-7936058aa1e4" (UID: "572f9e0c-4664-4a36-a52f-7936058aa1e4"). InnerVolumeSpecName "kube-api-access-qlb9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:54.898702 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.898668 5815 generic.go:356] "Generic (PLEG): container finished" podID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerID="2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122" exitCode=0 Nov 03 21:03:54.898841 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.898711 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" Nov 03 21:03:54.898841 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.898757 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" event={"ID":"c693e7b8-e44b-4fd8-843e-0a686c5fcce2","Type":"ContainerDied","Data":"2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122"} Nov 03 21:03:54.898841 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.898813 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6" event={"ID":"c693e7b8-e44b-4fd8-843e-0a686c5fcce2","Type":"ContainerDied","Data":"015665e8d2dcafb79fe3f1206201067482779041f937f1cfbf35ee3149f07210"} Nov 03 21:03:54.899053 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.898843 5815 scope.go:122] "RemoveContainer" containerID="2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122" Nov 03 21:03:54.901199 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.901173 5815 generic.go:356] "Generic (PLEG): container finished" podID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerID="631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21" exitCode=0 Nov 03 21:03:54.901321 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.901225 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" Nov 03 21:03:54.901321 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.901244 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" event={"ID":"572f9e0c-4664-4a36-a52f-7936058aa1e4","Type":"ContainerDied","Data":"631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21"} Nov 03 21:03:54.901321 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.901280 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47" event={"ID":"572f9e0c-4664-4a36-a52f-7936058aa1e4","Type":"ContainerDied","Data":"516b43d481f8683564c432f0a350f3a90328a0c5d5f2314adf00a2e1c112afd9"} Nov 03 21:03:54.913554 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.913539 5815 scope.go:122] "RemoveContainer" containerID="2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122" Nov 03 21:03:54.913824 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:54.913803 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122\": container with ID starting with 2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122 not found: ID does not exist" containerID="2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122" Nov 03 21:03:54.913922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.913830 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122"} err="failed to get container status \"2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122\": rpc error: code = NotFound desc = could not find container \"2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122\": container with ID starting with 2be3a98601e43654f7c902b66b03e68d117869851caf1d2a621c3b1db5481122 not found: ID does not exist" Nov 03 21:03:54.913922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.913848 5815 scope.go:122] "RemoveContainer" containerID="631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21" Nov 03 21:03:54.925667 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.925649 5815 scope.go:122] "RemoveContainer" containerID="631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21" Nov 03 21:03:54.925970 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:54.925942 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21\": container with ID starting with 631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21 not found: ID does not exist" containerID="631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21" Nov 03 21:03:54.926055 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.925982 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21"} err="failed to get container status \"631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21\": rpc error: code = NotFound desc = could not find container \"631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21\": container with ID starting with 631e7e9ca7f428cc93de77824a78436d7654e8d703d71b1b4c715417fc2dbf21 not found: ID does not exist" Nov 03 21:03:54.927128 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.927090 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6"] Nov 03 21:03:54.930477 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.930459 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3bcc2b-55b9879487-nxpb6"] Nov 03 21:03:54.942700 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.942680 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47"] Nov 03 21:03:54.947756 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.947737 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-3bcc2b-6dfd5ddd8d-9lx47"] Nov 03 21:03:54.990374 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.990346 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlb9q\" (UniqueName: \"kubernetes.io/projected/572f9e0c-4664-4a36-a52f-7936058aa1e4-kube-api-access-qlb9q\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:54.990374 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:54.990373 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572f9e0c-4664-4a36-a52f-7936058aa1e4-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:56.598530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:56.597643 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" path="/var/lib/kubelet/pods/572f9e0c-4664-4a36-a52f-7936058aa1e4/volumes" Nov 03 21:03:56.598530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:56.598425 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" path="/var/lib/kubelet/pods/c693e7b8-e44b-4fd8-843e-0a686c5fcce2/volumes" Nov 03 21:03:57.447662 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:57.447604 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:58.547497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.547474 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 21:03:58.637253 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637176 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637253 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637214 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637253 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637241 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637269 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637323 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637389 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637444 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637664 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637533 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:58.637664 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637576 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.637664 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637633 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy\") pod \"e2041c14-e13f-4e68-8008-df150d920310\" (UID: \"e2041c14-e13f-4e68-8008-df150d920310\") " Nov 03 21:03:58.638132 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.637830 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:58.638132 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638008 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:58.638132 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638034 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:58.638132 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638081 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:58.638409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638190 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.638409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638213 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.638409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638227 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2041c14-e13f-4e68-8008-df150d920310-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.638409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638240 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.638409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.638254 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.640289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.640262 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:58.640478 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.640457 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "kube-api-access-pf7cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:58.640679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.640659 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:58.640757 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.640716 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy" pod "e2041c14-e13f-4e68-8008-df150d920310" (UID: "e2041c14-e13f-4e68-8008-df150d920310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:58.739163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.739131 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2041c14-e13f-4e68-8008-df150d920310-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.739163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.739157 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2041c14-e13f-4e68-8008-df150d920310-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.739163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.739166 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf7cc\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-kube-api-access-pf7cc\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.739445 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.739177 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2041c14-e13f-4e68-8008-df150d920310-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:58.956269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.956179 5815 generic.go:356] "Generic (PLEG): container finished" podID="e2041c14-e13f-4e68-8008-df150d920310" containerID="ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53" exitCode=0 Nov 03 21:03:58.956269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.956240 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" Nov 03 21:03:58.956560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.956268 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" event={"ID":"e2041c14-e13f-4e68-8008-df150d920310","Type":"ContainerDied","Data":"ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53"} Nov 03 21:03:58.956560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.956325 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh" event={"ID":"e2041c14-e13f-4e68-8008-df150d920310","Type":"ContainerDied","Data":"bf4abce38065a39b3379fd0387d37208de3e07978c48a5a724e680de31963edb"} Nov 03 21:03:58.956560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.956337 5815 scope.go:122] "RemoveContainer" containerID="ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53" Nov 03 21:03:58.968238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.968220 5815 scope.go:122] "RemoveContainer" containerID="fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4" Nov 03 21:03:58.980818 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.980786 5815 scope.go:122] "RemoveContainer" containerID="ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53" Nov 03 21:03:58.981140 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:58.981115 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53\": container with ID starting with ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53 not found: ID does not exist" containerID="ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53" Nov 03 21:03:58.981232 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.981154 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53"} err="failed to get container status \"ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53\": rpc error: code = NotFound desc = could not find container \"ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53\": container with ID starting with ec6a255b7333639c82a4f7c8b21a54b388bd8f03de247e16e1a8858424d58e53 not found: ID does not exist" Nov 03 21:03:58.981232 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.981181 5815 scope.go:122] "RemoveContainer" containerID="fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4" Nov 03 21:03:58.981489 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:03:58.981465 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4\": container with ID starting with fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4 not found: ID does not exist" containerID="fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4" Nov 03 21:03:58.981583 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.981497 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4"} err="failed to get container status \"fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4\": rpc error: code = NotFound desc = could not find container \"fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4\": container with ID starting with fc6130de97290e487124f00a923c269b661a69447087582f7c576686d29c6ac4 not found: ID does not exist" Nov 03 21:03:58.983751 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.983309 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh"] Nov 03 21:03:58.986864 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:58.986835 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-5mpvh"] Nov 03 21:03:59.823893 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.823855 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr"] Nov 03 21:03:59.824622 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824600 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" Nov 03 21:03:59.824622 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824623 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824637 5815 cpu_manager.go:414] "containerMap: removing container" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerName="payload-processing" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824644 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerName="payload-processing" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824666 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e3057a3a-a231-4340-8752-d6d7b670f007" containerName="payload-processing" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824675 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3057a3a-a231-4340-8752-d6d7b670f007" containerName="payload-processing" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824693 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824699 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824716 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3c0db79b-4963-47ec-897b-d3248f62d373" containerName="revoke-keys" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824723 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0db79b-4963-47ec-897b-d3248f62d373" containerName="revoke-keys" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824744 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerName="payload-pre-processing" Nov 03 21:03:59.824817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824752 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerName="payload-pre-processing" Nov 03 21:03:59.825353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824884 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c693e7b8-e44b-4fd8-843e-0a686c5fcce2" containerName="payload-pre-processing" Nov 03 21:03:59.825353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824905 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" Nov 03 21:03:59.825353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824915 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="572f9e0c-4664-4a36-a52f-7936058aa1e4" containerName="payload-processing" Nov 03 21:03:59.825353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824927 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3057a3a-a231-4340-8752-d6d7b670f007" containerName="payload-processing" Nov 03 21:03:59.825353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.824940 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0db79b-4963-47ec-897b-d3248f62d373" containerName="revoke-keys" Nov 03 21:03:59.830539 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.830486 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:03:59.832900 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.832867 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr"] Nov 03 21:03:59.844524 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.841850 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:59.844524 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.843203 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:03:59.871449 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.871417 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:03:59.958351 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.958312 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:03:59.958884 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.958855 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:03:59.959095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:03:59.959077 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.059908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.059876 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.060076 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.059917 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.060076 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.059989 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.060679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.060655 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.062762 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.062738 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.076148 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.076090 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv\") pod \"maas-api-revoke-sub-45fb1561-8hdmr\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.087124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.087094 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq"] Nov 03 21:04:00.160803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.160772 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:00.453454 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.453411 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr"] Nov 03 21:04:00.454587 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:04:00.454563 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode903e189_38be_4525_a3f2_7cbf8a6f8503.slice/crio-de7cfc033e819018dae7934816742b2933ba88b77889ad1d8434622fea150a02 WatchSource:0}: Error finding container de7cfc033e819018dae7934816742b2933ba88b77889ad1d8434622fea150a02: Status 404 returned error can't find the container with id de7cfc033e819018dae7934816742b2933ba88b77889ad1d8434622fea150a02 Nov 03 21:04:00.584721 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.584680 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2041c14-e13f-4e68-8008-df150d920310" path="/var/lib/kubelet/pods/e2041c14-e13f-4e68-8008-df150d920310/volumes" Nov 03 21:04:00.979159 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.979114 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" event={"ID":"e903e189-38be-4525-a3f2-7cbf8a6f8503","Type":"ContainerStarted","Data":"3c14af120fc3b1f8af361eccd467f9d9a7a2e22a16c68d3f82f80b90448c23a3"} Nov 03 21:04:00.979655 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.979182 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" event={"ID":"e903e189-38be-4525-a3f2-7cbf8a6f8503","Type":"ContainerStarted","Data":"de7cfc033e819018dae7934816742b2933ba88b77889ad1d8434622fea150a02"} Nov 03 21:04:00.998416 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:00.998370 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" podStartSLOduration=1.998358263 podStartE2EDuration="1.998358263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:00.9966325 +0000 UTC m=+3750.526978174" watchObservedRunningTime="2026-11-03 21:04:00.998358263 +0000 UTC m=+3750.528703935" Nov 03 21:04:01.989806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:01.989760 5815 generic.go:356] "Generic (PLEG): container finished" podID="e903e189-38be-4525-a3f2-7cbf8a6f8503" containerID="3c14af120fc3b1f8af361eccd467f9d9a7a2e22a16c68d3f82f80b90448c23a3" exitCode=0 Nov 03 21:04:01.990232 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:01.989812 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" event={"ID":"e903e189-38be-4525-a3f2-7cbf8a6f8503","Type":"ContainerDied","Data":"3c14af120fc3b1f8af361eccd467f9d9a7a2e22a16c68d3f82f80b90448c23a3"} Nov 03 21:04:03.159705 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.159681 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:03.308828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.308752 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv\") pod \"e903e189-38be-4525-a3f2-7cbf8a6f8503\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " Nov 03 21:04:03.308828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.308797 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca\") pod \"e903e189-38be-4525-a3f2-7cbf8a6f8503\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " Nov 03 21:04:03.309063 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.308869 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca\") pod \"e903e189-38be-4525-a3f2-7cbf8a6f8503\" (UID: \"e903e189-38be-4525-a3f2-7cbf8a6f8503\") " Nov 03 21:04:03.309973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.309847 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca" pod "e903e189-38be-4525-a3f2-7cbf8a6f8503" (UID: "e903e189-38be-4525-a3f2-7cbf8a6f8503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:03.311266 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.311241 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv" pod "e903e189-38be-4525-a3f2-7cbf8a6f8503" (UID: "e903e189-38be-4525-a3f2-7cbf8a6f8503"). InnerVolumeSpecName "kube-api-access-965sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:03.311536 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.311482 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca" pod "e903e189-38be-4525-a3f2-7cbf8a6f8503" (UID: "e903e189-38be-4525-a3f2-7cbf8a6f8503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:03.410742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.410705 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-965sv\" (UniqueName: \"kubernetes.io/projected/e903e189-38be-4525-a3f2-7cbf8a6f8503-kube-api-access-965sv\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:03.410742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.410734 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:03.410742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:03.410745 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e903e189-38be-4525-a3f2-7cbf8a6f8503-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:04.006953 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:04.006917 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" Nov 03 21:04:04.007201 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:04.006922 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr" event={"ID":"e903e189-38be-4525-a3f2-7cbf8a6f8503","Type":"ContainerDied","Data":"de7cfc033e819018dae7934816742b2933ba88b77889ad1d8434622fea150a02"} Nov 03 21:04:04.007201 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:04.007056 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7cfc033e819018dae7934816742b2933ba88b77889ad1d8434622fea150a02" Nov 03 21:04:05.087860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.087811 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" podUID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" containerName="payload-pre-processing" containerID="cri-o://8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040" gracePeriod=25 Nov 03 21:04:05.360331 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.360309 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 21:04:05.441541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.441496 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs\" (UniqueName: \"kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs\") pod \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " Nov 03 21:04:05.441723 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.441689 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume\") pod \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\" (UID: \"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180\") " Nov 03 21:04:05.442014 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.441993 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume" pod "99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" (UID: "99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:05.443934 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.443903 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs" pod "99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" (UID: "99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180"). InnerVolumeSpecName "kube-api-access-98qzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:05.544278 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.543030 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98qzs\" (UniqueName: \"kubernetes.io/projected/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-kube-api-access-98qzs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:05.544278 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:05.543071 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:06.032803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.030579 5815 generic.go:356] "Generic (PLEG): container finished" podID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" containerID="8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040" exitCode=0 Nov 03 21:04:06.032803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.030645 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" Nov 03 21:04:06.032803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.030648 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" event={"ID":"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180","Type":"ContainerDied","Data":"8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040"} Nov 03 21:04:06.032803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.030727 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" event={"ID":"99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180","Type":"ContainerDied","Data":"984e2f0dbcaf0ff1865b22bd1e46a63fd87aa8e705ccf3d7a279ab866857e812"} Nov 03 21:04:06.032803 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.030758 5815 scope.go:122] "RemoveContainer" containerID="8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040" Nov 03 21:04:06.052191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.052173 5815 scope.go:122] "RemoveContainer" containerID="8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040" Nov 03 21:04:06.052500 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:06.052477 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040\": container with ID starting with 8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040 not found: ID does not exist" containerID="8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040" Nov 03 21:04:06.052606 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.052541 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040"} err="failed to get container status \"8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040\": rpc error: code = NotFound desc = could not find container \"8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040\": container with ID starting with 8fa6ff735f8b9c5462d6c4dd2487c7e9c02f11e71019de0575d1f73cf1a5b040 not found: ID does not exist" Nov 03 21:04:06.058941 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.058917 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq"] Nov 03 21:04:06.064011 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.063992 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq"] Nov 03 21:04:06.274466 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.274421 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c7aecb-6868956579-h9vjq" podUID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.22:50052\" within 1s: context deadline exceeded" Nov 03 21:04:06.274930 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:04:06.274501 5815 logging.go:55] [core] [Channel #10580 SubChannel #10581]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.22:50052", ServerName: "10.132.1.22:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.22:50052: operation was canceled" Nov 03 21:04:06.585310 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:06.585278 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" path="/var/lib/kubelet/pods/99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180/volumes" Nov 03 21:04:07.447306 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:07.447266 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:04:10.100148 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.099935 5815 generic.go:356] "Generic (PLEG): container finished" podID="959d9d63-751d-4551-a9d0-8b7f50625066" containerID="54987fae2140f60b231ae3ec200c1cf00382f0672104e18342468afc91e65023" exitCode=0 Nov 03 21:04:10.100148 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.100081 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" event={"ID":"959d9d63-751d-4551-a9d0-8b7f50625066","Type":"ContainerDied","Data":"54987fae2140f60b231ae3ec200c1cf00382f0672104e18342468afc91e65023"} Nov 03 21:04:10.145684 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.145658 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 21:04:10.307774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.307740 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp\" (UniqueName: \"kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp\") pod \"959d9d63-751d-4551-a9d0-8b7f50625066\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " Nov 03 21:04:10.307999 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.307959 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume\") pod \"959d9d63-751d-4551-a9d0-8b7f50625066\" (UID: \"959d9d63-751d-4551-a9d0-8b7f50625066\") " Nov 03 21:04:10.308425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.308387 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume" pod "959d9d63-751d-4551-a9d0-8b7f50625066" (UID: "959d9d63-751d-4551-a9d0-8b7f50625066"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:10.310387 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.310361 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp" pod "959d9d63-751d-4551-a9d0-8b7f50625066" (UID: "959d9d63-751d-4551-a9d0-8b7f50625066"). InnerVolumeSpecName "kube-api-access-99fzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:10.376388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.376288 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns"] Nov 03 21:04:10.409308 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.409270 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959d9d63-751d-4551-a9d0-8b7f50625066-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:10.409308 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.409310 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fzp\" (UniqueName: \"kubernetes.io/projected/959d9d63-751d-4551-a9d0-8b7f50625066-kube-api-access-99fzp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:10.844375 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844341 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88"] Nov 03 21:04:10.844847 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844834 5815 cpu_manager.go:414] "containerMap: removing container" podUID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" containerName="payload-pre-processing" Nov 03 21:04:10.844903 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844849 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" containerName="payload-pre-processing" Nov 03 21:04:10.844903 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844865 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e903e189-38be-4525-a3f2-7cbf8a6f8503" containerName="revoke-keys" Nov 03 21:04:10.844903 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844870 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e903e189-38be-4525-a3f2-7cbf8a6f8503" containerName="revoke-keys" Nov 03 21:04:10.844903 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844886 5815 cpu_manager.go:414] "containerMap: removing container" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" containerName="payload-pre-processing" Nov 03 21:04:10.844903 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844892 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" containerName="payload-pre-processing" Nov 03 21:04:10.845078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844969 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="99cd0dc4-6ccd-4bf7-b158-5dcb1fb7f180" containerName="payload-pre-processing" Nov 03 21:04:10.845078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844980 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e903e189-38be-4525-a3f2-7cbf8a6f8503" containerName="revoke-keys" Nov 03 21:04:10.845078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844989 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" containerName="payload-pre-processing" Nov 03 21:04:10.845078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.844998 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2041c14-e13f-4e68-8008-df150d920310" containerName="istio-proxy" Nov 03 21:04:10.848447 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.848425 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:10.856530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.854282 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:04:10.856530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.854397 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:04:10.858641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:10.858615 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88"] Nov 03 21:04:11.017825 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.017789 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.018013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.017991 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77dq\" (UniqueName: \"kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.018093 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.018030 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.110742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.110642 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" Nov 03 21:04:11.110742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.110658 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq" event={"ID":"959d9d63-751d-4551-a9d0-8b7f50625066","Type":"ContainerDied","Data":"44fe0f8bd420ffc6e9cec9a1ad06c3e48a998ad2d0098f88c86b56564597ab92"} Nov 03 21:04:11.110742 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.110726 5815 scope.go:122] "RemoveContainer" containerID="54987fae2140f60b231ae3ec200c1cf00382f0672104e18342468afc91e65023" Nov 03 21:04:11.118996 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.118966 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77dq\" (UniqueName: \"kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.119156 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.119004 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.119156 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.119069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.119804 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.119778 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.121710 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.121686 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.130935 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.130900 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq"] Nov 03 21:04:11.134069 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.134042 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77dq\" (UniqueName: \"kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq\") pod \"maas-api-revoke-sub-23e031ee-s6z88\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.138806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.138782 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-37aa95-f58cfd595-9vdqq"] Nov 03 21:04:11.197047 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.197010 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:11.493539 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:11.492892 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88"] Nov 03 21:04:11.496539 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:04:11.496481 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa2e081_bd00_492f_b3f7_8c0d537f5744.slice/crio-0cd182acd40a5e29c6635cd47acd39b4919d9c20ed53bbf945daa69e2cb153b1 WatchSource:0}: Error finding container 0cd182acd40a5e29c6635cd47acd39b4919d9c20ed53bbf945daa69e2cb153b1: Status 404 returned error can't find the container with id 0cd182acd40a5e29c6635cd47acd39b4919d9c20ed53bbf945daa69e2cb153b1 Nov 03 21:04:12.120928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:12.120844 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" event={"ID":"aaa2e081-bd00-492f-b3f7-8c0d537f5744","Type":"ContainerStarted","Data":"3dbabf118d8d04e34e8686f57d506e6675517bda934b926ff8276eacb6720e89"} Nov 03 21:04:12.120928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:12.120922 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" event={"ID":"aaa2e081-bd00-492f-b3f7-8c0d537f5744","Type":"ContainerStarted","Data":"0cd182acd40a5e29c6635cd47acd39b4919d9c20ed53bbf945daa69e2cb153b1"} Nov 03 21:04:12.144823 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:12.144774 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" podStartSLOduration=2.144762164 podStartE2EDuration="2.144762164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:12.141861113 +0000 UTC m=+3761.672206806" watchObservedRunningTime="2026-11-03 21:04:12.144762164 +0000 UTC m=+3761.675107835" Nov 03 21:04:12.586182 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:12.586143 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959d9d63-751d-4551-a9d0-8b7f50625066" path="/var/lib/kubelet/pods/959d9d63-751d-4551-a9d0-8b7f50625066/volumes" Nov 03 21:04:13.130115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:13.130073 5815 generic.go:356] "Generic (PLEG): container finished" podID="aaa2e081-bd00-492f-b3f7-8c0d537f5744" containerID="3dbabf118d8d04e34e8686f57d506e6675517bda934b926ff8276eacb6720e89" exitCode=0 Nov 03 21:04:13.130613 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:13.130164 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" event={"ID":"aaa2e081-bd00-492f-b3f7-8c0d537f5744","Type":"ContainerDied","Data":"3dbabf118d8d04e34e8686f57d506e6675517bda934b926ff8276eacb6720e89"} Nov 03 21:04:13.871035 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:13.870995 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:04:14.296832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.296809 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:14.365302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.365271 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca\") pod \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " Nov 03 21:04:14.365482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.365375 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca\") pod \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " Nov 03 21:04:14.365482 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.365440 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq\" (UniqueName: \"kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq\") pod \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\" (UID: \"aaa2e081-bd00-492f-b3f7-8c0d537f5744\") " Nov 03 21:04:14.365641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.365624 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca" pod "aaa2e081-bd00-492f-b3f7-8c0d537f5744" (UID: "aaa2e081-bd00-492f-b3f7-8c0d537f5744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:14.365846 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.365821 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:14.367881 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.367848 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca" pod "aaa2e081-bd00-492f-b3f7-8c0d537f5744" (UID: "aaa2e081-bd00-492f-b3f7-8c0d537f5744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:14.367972 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.367927 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq" pod "aaa2e081-bd00-492f-b3f7-8c0d537f5744" (UID: "aaa2e081-bd00-492f-b3f7-8c0d537f5744"). InnerVolumeSpecName "kube-api-access-h77dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:14.466405 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.466376 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa2e081-bd00-492f-b3f7-8c0d537f5744-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:14.466405 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:14.466406 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h77dq\" (UniqueName: \"kubernetes.io/projected/aaa2e081-bd00-492f-b3f7-8c0d537f5744-kube-api-access-h77dq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:15.147359 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:15.147327 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" Nov 03 21:04:15.147359 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:15.147346 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88" event={"ID":"aaa2e081-bd00-492f-b3f7-8c0d537f5744","Type":"ContainerDied","Data":"0cd182acd40a5e29c6635cd47acd39b4919d9c20ed53bbf945daa69e2cb153b1"} Nov 03 21:04:15.147644 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:15.147403 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd182acd40a5e29c6635cd47acd39b4919d9c20ed53bbf945daa69e2cb153b1" Nov 03 21:04:15.420863 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:15.420772 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47"] Nov 03 21:04:18.784000 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:18.783949 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:04:20.291378 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:20.291343 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp"] Nov 03 21:04:20.306832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:20.306802 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g"] Nov 03 21:04:25.292586 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.292501 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" podUID="69967678-8d12-435a-8d46-341b7485f73e" containerName="payload-processing" containerID="cri-o://b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9" gracePeriod=25 Nov 03 21:04:25.307249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.307212 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerName="payload-pre-processing" containerID="cri-o://64fa0082267425f332563ef26109b6d3538de732cc5de856d1c3bd85d705d11e" gracePeriod=25 Nov 03 21:04:25.397131 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.397081 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" podUID="72f29c2b-2da7-45b6-bace-92b83127a214" containerName="main" containerID="cri-o://684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828" gracePeriod=45 Nov 03 21:04:25.576264 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.576236 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 21:04:25.687968 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.687940 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl\" (UniqueName: \"kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl\") pod \"69967678-8d12-435a-8d46-341b7485f73e\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " Nov 03 21:04:25.688282 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.688257 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume\") pod \"69967678-8d12-435a-8d46-341b7485f73e\" (UID: \"69967678-8d12-435a-8d46-341b7485f73e\") " Nov 03 21:04:25.688699 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.688672 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume" pod "69967678-8d12-435a-8d46-341b7485f73e" (UID: "69967678-8d12-435a-8d46-341b7485f73e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:25.689178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.689156 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69967678-8d12-435a-8d46-341b7485f73e-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.695618 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.695588 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl" pod "69967678-8d12-435a-8d46-341b7485f73e" (UID: "69967678-8d12-435a-8d46-341b7485f73e"). InnerVolumeSpecName "kube-api-access-7tmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:25.697283 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.697261 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:04:25.790239 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790209 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs\" (UniqueName: \"kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs\") pod \"72f29c2b-2da7-45b6-bace-92b83127a214\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " Nov 03 21:04:25.790239 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790245 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache\") pod \"72f29c2b-2da7-45b6-bace-92b83127a214\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " Nov 03 21:04:25.790541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790311 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir\") pod \"72f29c2b-2da7-45b6-bace-92b83127a214\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " Nov 03 21:04:25.790541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790332 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home\") pod \"72f29c2b-2da7-45b6-bace-92b83127a214\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " Nov 03 21:04:25.790541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790418 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h\") pod \"72f29c2b-2da7-45b6-bace-92b83127a214\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " Nov 03 21:04:25.790735 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790553 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm\") pod \"72f29c2b-2da7-45b6-bace-92b83127a214\" (UID: \"72f29c2b-2da7-45b6-bace-92b83127a214\") " Nov 03 21:04:25.790735 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790657 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache" pod "72f29c2b-2da7-45b6-bace-92b83127a214" (UID: "72f29c2b-2da7-45b6-bace-92b83127a214"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:25.790735 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790671 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir" pod "72f29c2b-2da7-45b6-bace-92b83127a214" (UID: "72f29c2b-2da7-45b6-bace-92b83127a214"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:25.791018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790954 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tmxl\" (UniqueName: \"kubernetes.io/projected/69967678-8d12-435a-8d46-341b7485f73e-kube-api-access-7tmxl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.791018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790981 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.791018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.790996 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.791305 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.791093 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home" pod "72f29c2b-2da7-45b6-bace-92b83127a214" (UID: "72f29c2b-2da7-45b6-bace-92b83127a214"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:25.793049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.793015 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm" pod "72f29c2b-2da7-45b6-bace-92b83127a214" (UID: "72f29c2b-2da7-45b6-bace-92b83127a214"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:25.793522 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.793489 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs" pod "72f29c2b-2da7-45b6-bace-92b83127a214" (UID: "72f29c2b-2da7-45b6-bace-92b83127a214"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:25.793621 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.793564 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h" pod "72f29c2b-2da7-45b6-bace-92b83127a214" (UID: "72f29c2b-2da7-45b6-bace-92b83127a214"). InnerVolumeSpecName "kube-api-access-csn4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:25.891914 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.891841 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.891914 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.891869 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f29c2b-2da7-45b6-bace-92b83127a214-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.891914 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.891884 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f29c2b-2da7-45b6-bace-92b83127a214-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:25.891914 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:25.891894 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csn4h\" (UniqueName: \"kubernetes.io/projected/72f29c2b-2da7-45b6-bace-92b83127a214-kube-api-access-csn4h\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:26.254369 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.254332 5815 generic.go:356] "Generic (PLEG): container finished" podID="69967678-8d12-435a-8d46-341b7485f73e" containerID="b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9" exitCode=0 Nov 03 21:04:26.254567 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.254375 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" Nov 03 21:04:26.254567 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.254421 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" event={"ID":"69967678-8d12-435a-8d46-341b7485f73e","Type":"ContainerDied","Data":"b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9"} Nov 03 21:04:26.254567 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.254473 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp" event={"ID":"69967678-8d12-435a-8d46-341b7485f73e","Type":"ContainerDied","Data":"1e0f0c0b95b6ed9dfa78143ec90e912aa5f9adece3bd145918cbd415ea8d7451"} Nov 03 21:04:26.254567 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.254518 5815 scope.go:122] "RemoveContainer" containerID="b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9" Nov 03 21:04:26.256409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.256383 5815 generic.go:356] "Generic (PLEG): container finished" podID="72f29c2b-2da7-45b6-bace-92b83127a214" containerID="684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828" exitCode=0 Nov 03 21:04:26.256529 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.256418 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" Nov 03 21:04:26.256529 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.256438 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" event={"ID":"72f29c2b-2da7-45b6-bace-92b83127a214","Type":"ContainerDied","Data":"684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828"} Nov 03 21:04:26.256529 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.256467 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns" event={"ID":"72f29c2b-2da7-45b6-bace-92b83127a214","Type":"ContainerDied","Data":"a162b571225996550fc9628522f9a229223a66072b26c1678b5eb494d00be17d"} Nov 03 21:04:26.267113 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.267091 5815 scope.go:122] "RemoveContainer" containerID="b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9" Nov 03 21:04:26.267395 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:26.267373 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9\": container with ID starting with b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9 not found: ID does not exist" containerID="b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9" Nov 03 21:04:26.267471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.267403 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9"} err="failed to get container status \"b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9\": rpc error: code = NotFound desc = could not find container \"b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9\": container with ID starting with b2715eec630f33d4c3fdc6e72084b757d12c2425684e3dcf8da2a5d7845462b9 not found: ID does not exist" Nov 03 21:04:26.267471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.267420 5815 scope.go:122] "RemoveContainer" containerID="684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828" Nov 03 21:04:26.278102 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.278084 5815 scope.go:122] "RemoveContainer" containerID="684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828" Nov 03 21:04:26.278389 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:26.278368 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828\": container with ID starting with 684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828 not found: ID does not exist" containerID="684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828" Nov 03 21:04:26.278489 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.278395 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828"} err="failed to get container status \"684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828\": rpc error: code = NotFound desc = could not find container \"684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828\": container with ID starting with 684c9f28872fee50bdec3642660b81802a9589dee93d252a96f7ae454c6ca828 not found: ID does not exist" Nov 03 21:04:26.283369 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.283347 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns"] Nov 03 21:04:26.291257 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.291236 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ba89e0/sub-test-model-ba89e0-kserve-cb5d59d8c-9qnns"] Nov 03 21:04:26.302868 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.302836 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp"] Nov 03 21:04:26.310644 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.310626 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-db5198-666565cf58-659tp"] Nov 03 21:04:26.587353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.587266 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69967678-8d12-435a-8d46-341b7485f73e" path="/var/lib/kubelet/pods/69967678-8d12-435a-8d46-341b7485f73e/volumes" Nov 03 21:04:26.587900 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:26.587878 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f29c2b-2da7-45b6-bace-92b83127a214" path="/var/lib/kubelet/pods/72f29c2b-2da7-45b6-bace-92b83127a214/volumes" Nov 03 21:04:27.627348 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:27.627305 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:04:29.002557 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.002492 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp"] Nov 03 21:04:29.003223 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003197 5815 cpu_manager.go:414] "containerMap: removing container" podUID="69967678-8d12-435a-8d46-341b7485f73e" containerName="payload-processing" Nov 03 21:04:29.003223 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003225 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69967678-8d12-435a-8d46-341b7485f73e" containerName="payload-processing" Nov 03 21:04:29.003393 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003244 5815 cpu_manager.go:414] "containerMap: removing container" podUID="aaa2e081-bd00-492f-b3f7-8c0d537f5744" containerName="revoke-keys" Nov 03 21:04:29.003393 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003252 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa2e081-bd00-492f-b3f7-8c0d537f5744" containerName="revoke-keys" Nov 03 21:04:29.003393 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003295 5815 cpu_manager.go:414] "containerMap: removing container" podUID="72f29c2b-2da7-45b6-bace-92b83127a214" containerName="main" Nov 03 21:04:29.003393 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003301 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f29c2b-2da7-45b6-bace-92b83127a214" containerName="main" Nov 03 21:04:29.003580 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003407 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="69967678-8d12-435a-8d46-341b7485f73e" containerName="payload-processing" Nov 03 21:04:29.003580 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003419 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa2e081-bd00-492f-b3f7-8c0d537f5744" containerName="revoke-keys" Nov 03 21:04:29.003580 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.003435 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f29c2b-2da7-45b6-bace-92b83127a214" containerName="main" Nov 03 21:04:29.008441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.008424 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.013457 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.013175 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:04:29.013457 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.013305 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:04:29.016329 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.016307 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp"] Nov 03 21:04:29.035530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.029467 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.035530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.029564 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2khlk\" (UniqueName: \"kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.035530 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.029618 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.132746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.132701 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.133613 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.133585 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2khlk\" (UniqueName: \"kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.133818 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.133796 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.138214 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.138170 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.139532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.139483 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.152947 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.152921 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2khlk\" (UniqueName: \"kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk\") pod \"maas-api-revoke-sub-a1b8ba30-ghbvp\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.326237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.326151 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:29.613860 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:04:29.613829 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341243bf_d4ba_447f_8317_326900c3535e.slice/crio-7343bf8ed9bc55bfbb76f24c0f14032f505c455031dec1e3e5b3d68c28889742 WatchSource:0}: Error finding container 7343bf8ed9bc55bfbb76f24c0f14032f505c455031dec1e3e5b3d68c28889742: Status 404 returned error can't find the container with id 7343bf8ed9bc55bfbb76f24c0f14032f505c455031dec1e3e5b3d68c28889742 Nov 03 21:04:29.614434 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:29.614401 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp"] Nov 03 21:04:30.296252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.296211 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" event={"ID":"341243bf-d4ba-447f-8317-326900c3535e","Type":"ContainerStarted","Data":"dbe9b688bf3960b74433f2355fe4627f94a38ace67d934de10c259dcdacd8da1"} Nov 03 21:04:30.296860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.296276 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" event={"ID":"341243bf-d4ba-447f-8317-326900c3535e","Type":"ContainerStarted","Data":"7343bf8ed9bc55bfbb76f24c0f14032f505c455031dec1e3e5b3d68c28889742"} Nov 03 21:04:30.321987 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.321923 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" podStartSLOduration=2.321903599 podStartE2EDuration="2.321903599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:30.315987709 +0000 UTC m=+3779.846333383" watchObservedRunningTime="2026-11-03 21:04:30.321903599 +0000 UTC m=+3779.852249271" Nov 03 21:04:30.442438 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.442389 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" containerID="cri-o://3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e" gracePeriod=45 Nov 03 21:04:30.751435 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.751406 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 21:04:30.850901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.850818 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache\") pod \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " Nov 03 21:04:30.850901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.850871 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm\") pod \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " Nov 03 21:04:30.850901 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.850898 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir\") pod \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " Nov 03 21:04:30.851175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.850924 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home\") pod \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " Nov 03 21:04:30.851175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851083 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5\" (UniqueName: \"kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5\") pod \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " Nov 03 21:04:30.851175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851129 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs\" (UniqueName: \"kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs\") pod \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\" (UID: \"12da41d9-bf1f-4cd6-9993-ba7234f4c732\") " Nov 03 21:04:30.851369 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851145 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache" pod "12da41d9-bf1f-4cd6-9993-ba7234f4c732" (UID: "12da41d9-bf1f-4cd6-9993-ba7234f4c732"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.851369 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851212 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir" pod "12da41d9-bf1f-4cd6-9993-ba7234f4c732" (UID: "12da41d9-bf1f-4cd6-9993-ba7234f4c732"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.851369 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851320 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home" pod "12da41d9-bf1f-4cd6-9993-ba7234f4c732" (UID: "12da41d9-bf1f-4cd6-9993-ba7234f4c732"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.851604 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851566 5815 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-home\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.851604 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851600 5815 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-model-cache\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.851604 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.851616 5815 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tmp-dir\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.853490 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.853465 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm" pod "12da41d9-bf1f-4cd6-9993-ba7234f4c732" (UID: "12da41d9-bf1f-4cd6-9993-ba7234f4c732"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.853665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.853489 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs" pod "12da41d9-bf1f-4cd6-9993-ba7234f4c732" (UID: "12da41d9-bf1f-4cd6-9993-ba7234f4c732"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:30.853665 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.853543 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5" pod "12da41d9-bf1f-4cd6-9993-ba7234f4c732" (UID: "12da41d9-bf1f-4cd6-9993-ba7234f4c732"). InnerVolumeSpecName "kube-api-access-54bh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:30.952162 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.952139 5815 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12da41d9-bf1f-4cd6-9993-ba7234f4c732-tls-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.952162 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.952161 5815 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12da41d9-bf1f-4cd6-9993-ba7234f4c732-dshm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.952383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:30.952172 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bh5\" (UniqueName: \"kubernetes.io/projected/12da41d9-bf1f-4cd6-9993-ba7234f4c732-kube-api-access-54bh5\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:31.306337 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.306300 5815 generic.go:356] "Generic (PLEG): container finished" podID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerID="3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e" exitCode=0 Nov 03 21:04:31.306838 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.306365 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" Nov 03 21:04:31.306838 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.306439 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" event={"ID":"12da41d9-bf1f-4cd6-9993-ba7234f4c732","Type":"ContainerDied","Data":"3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e"} Nov 03 21:04:31.306838 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.306479 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47" event={"ID":"12da41d9-bf1f-4cd6-9993-ba7234f4c732","Type":"ContainerDied","Data":"7acf1185f52a362f794a3c04bfe95b7d9983066b01ff4beb2c2b8001622a2b98"} Nov 03 21:04:31.329888 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.328173 5815 scope.go:122] "RemoveContainer" containerID="3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e" Nov 03 21:04:31.334054 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.333756 5815 generic.go:356] "Generic (PLEG): container finished" podID="341243bf-d4ba-447f-8317-326900c3535e" containerID="dbe9b688bf3960b74433f2355fe4627f94a38ace67d934de10c259dcdacd8da1" exitCode=0 Nov 03 21:04:31.334054 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.333796 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" event={"ID":"341243bf-d4ba-447f-8317-326900c3535e","Type":"ContainerDied","Data":"dbe9b688bf3960b74433f2355fe4627f94a38ace67d934de10c259dcdacd8da1"} Nov 03 21:04:31.345244 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.345216 5815 scope.go:122] "RemoveContainer" containerID="3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e" Nov 03 21:04:31.345571 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:31.345549 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e\": container with ID starting with 3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e not found: ID does not exist" containerID="3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e" Nov 03 21:04:31.345632 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.345582 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e"} err="failed to get container status \"3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e\": rpc error: code = NotFound desc = could not find container \"3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e\": container with ID starting with 3f4de68c8512dfd280ab16394790afe40048e343811dcc514916efe6477e092e not found: ID does not exist" Nov 03 21:04:31.367423 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.367379 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47"] Nov 03 21:04:31.374575 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:31.374546 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-2-w2-085566-kserve-69577677d-x8x47"] Nov 03 21:04:32.492316 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.492291 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:32.565383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.565351 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca\") pod \"341243bf-d4ba-447f-8317-326900c3535e\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " Nov 03 21:04:32.565603 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.565389 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca\") pod \"341243bf-d4ba-447f-8317-326900c3535e\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " Nov 03 21:04:32.565603 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.565453 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk\" (UniqueName: \"kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk\") pod \"341243bf-d4ba-447f-8317-326900c3535e\" (UID: \"341243bf-d4ba-447f-8317-326900c3535e\") " Nov 03 21:04:32.565793 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.565767 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca" pod "341243bf-d4ba-447f-8317-326900c3535e" (UID: "341243bf-d4ba-447f-8317-326900c3535e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:32.567820 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.567789 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca" pod "341243bf-d4ba-447f-8317-326900c3535e" (UID: "341243bf-d4ba-447f-8317-326900c3535e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:32.567820 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.567809 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk" pod "341243bf-d4ba-447f-8317-326900c3535e" (UID: "341243bf-d4ba-447f-8317-326900c3535e"). InnerVolumeSpecName "kube-api-access-2khlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:32.584864 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.584839 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" path="/var/lib/kubelet/pods/12da41d9-bf1f-4cd6-9993-ba7234f4c732/volumes" Nov 03 21:04:32.666609 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.666577 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341243bf-d4ba-447f-8317-326900c3535e-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:32.666741 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.666610 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2khlk\" (UniqueName: \"kubernetes.io/projected/341243bf-d4ba-447f-8317-326900c3535e-kube-api-access-2khlk\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:32.666741 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:32.666625 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341243bf-d4ba-447f-8317-326900c3535e-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.349398 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:33.349360 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" Nov 03 21:04:33.349619 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:33.349402 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp" event={"ID":"341243bf-d4ba-447f-8317-326900c3535e","Type":"ContainerDied","Data":"7343bf8ed9bc55bfbb76f24c0f14032f505c455031dec1e3e5b3d68c28889742"} Nov 03 21:04:33.349619 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:33.349463 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7343bf8ed9bc55bfbb76f24c0f14032f505c455031dec1e3e5b3d68c28889742" Nov 03 21:04:33.459136 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:33.459108 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:04:35.887021 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:35.886985 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj"] Nov 03 21:04:35.892688 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:35.892661 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r"] Nov 03 21:04:37.639441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:37.639398 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:04:40.888030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:40.887940 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" podUID="fa2a804a-a656-4bd8-803e-02007913f36d" containerName="payload-processing" containerID="cri-o://ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878" gracePeriod=25 Nov 03 21:04:40.893842 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:40.893803 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" podUID="e62cfde2-4364-452b-818c-43450e0ce767" containerName="payload-pre-processing" containerID="cri-o://fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37" gracePeriod=25 Nov 03 21:04:41.152344 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.152287 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 21:04:41.269860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.269821 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4\" (UniqueName: \"kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4\") pod \"fa2a804a-a656-4bd8-803e-02007913f36d\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " Nov 03 21:04:41.270070 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.270012 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume\") pod \"fa2a804a-a656-4bd8-803e-02007913f36d\" (UID: \"fa2a804a-a656-4bd8-803e-02007913f36d\") " Nov 03 21:04:41.270425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.270394 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume" pod "fa2a804a-a656-4bd8-803e-02007913f36d" (UID: "fa2a804a-a656-4bd8-803e-02007913f36d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:41.272236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.272201 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4" pod "fa2a804a-a656-4bd8-803e-02007913f36d" (UID: "fa2a804a-a656-4bd8-803e-02007913f36d"). InnerVolumeSpecName "kube-api-access-d6pk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:41.371173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.371129 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2a804a-a656-4bd8-803e-02007913f36d-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:41.371173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.371159 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6pk4\" (UniqueName: \"kubernetes.io/projected/fa2a804a-a656-4bd8-803e-02007913f36d-kube-api-access-d6pk4\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:41.415919 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.415835 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" Nov 03 21:04:41.415919 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.415854 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" event={"ID":"fa2a804a-a656-4bd8-803e-02007913f36d","Type":"ContainerDied","Data":"ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878"} Nov 03 21:04:41.416206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.415837 5815 generic.go:356] "Generic (PLEG): container finished" podID="fa2a804a-a656-4bd8-803e-02007913f36d" containerID="ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878" exitCode=0 Nov 03 21:04:41.416281 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.416206 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj" event={"ID":"fa2a804a-a656-4bd8-803e-02007913f36d","Type":"ContainerDied","Data":"41d8bdb92d2ee57668b0c2d052ce1ffb0a057dd00ca5c93f80c86f63cf29bcd1"} Nov 03 21:04:41.416281 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.415944 5815 scope.go:122] "RemoveContainer" containerID="ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878" Nov 03 21:04:41.429543 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.429490 5815 scope.go:122] "RemoveContainer" containerID="ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878" Nov 03 21:04:41.429846 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:41.429824 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878\": container with ID starting with ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878 not found: ID does not exist" containerID="ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878" Nov 03 21:04:41.429914 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.429854 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878"} err="failed to get container status \"ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878\": rpc error: code = NotFound desc = could not find container \"ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878\": container with ID starting with ba37a79860198fd1bfe67f1887be29ac434160302ccb6e0c2f5f4967afb3a878 not found: ID does not exist" Nov 03 21:04:41.446989 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.446965 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj"] Nov 03 21:04:41.450006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:41.449982 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-421604-85fcf48bcb-rjhqj"] Nov 03 21:04:42.585495 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:42.585461 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2a804a-a656-4bd8-803e-02007913f36d" path="/var/lib/kubelet/pods/fa2a804a-a656-4bd8-803e-02007913f36d/volumes" Nov 03 21:04:43.851030 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:04:43.850995 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:04:45.452944 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.452907 5815 generic.go:356] "Generic (PLEG): container finished" podID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerID="64fa0082267425f332563ef26109b6d3538de732cc5de856d1c3bd85d705d11e" exitCode=0 Nov 03 21:04:45.453353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.452986 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" event={"ID":"e514b625-a309-4e35-b49f-7fbdadf82c0b","Type":"ContainerDied","Data":"64fa0082267425f332563ef26109b6d3538de732cc5de856d1c3bd85d705d11e"} Nov 03 21:04:45.581609 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.581573 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 21:04:45.615620 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.615591 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume\") pod \"e514b625-a309-4e35-b49f-7fbdadf82c0b\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " Nov 03 21:04:45.615800 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.615682 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp\" (UniqueName: \"kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp\") pod \"e514b625-a309-4e35-b49f-7fbdadf82c0b\" (UID: \"e514b625-a309-4e35-b49f-7fbdadf82c0b\") " Nov 03 21:04:45.616204 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.615996 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume" pod "e514b625-a309-4e35-b49f-7fbdadf82c0b" (UID: "e514b625-a309-4e35-b49f-7fbdadf82c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:45.616204 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.616167 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e514b625-a309-4e35-b49f-7fbdadf82c0b-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:45.618314 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.618287 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp" pod "e514b625-a309-4e35-b49f-7fbdadf82c0b" (UID: "e514b625-a309-4e35-b49f-7fbdadf82c0b"). InnerVolumeSpecName "kube-api-access-997vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:45.717273 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:45.717238 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997vp\" (UniqueName: \"kubernetes.io/projected/e514b625-a309-4e35-b49f-7fbdadf82c0b-kube-api-access-997vp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:46.343224 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.343187 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" podUID="e62cfde2-4364-452b-818c-43450e0ce767" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:04:46.472809 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.472762 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" event={"ID":"e514b625-a309-4e35-b49f-7fbdadf82c0b","Type":"ContainerDied","Data":"14ce122dea1adbb046635c8707d82753724acfd85cc21e6e3db9e4d224c1e0ae"} Nov 03 21:04:46.472809 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.472798 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g" Nov 03 21:04:46.473407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.472878 5815 scope.go:122] "RemoveContainer" containerID="64fa0082267425f332563ef26109b6d3538de732cc5de856d1c3bd85d705d11e" Nov 03 21:04:46.501996 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.501964 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g"] Nov 03 21:04:46.506710 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.506688 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-db5198-6dd79d66c5-cjn4g"] Nov 03 21:04:46.585991 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:46.585950 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" path="/var/lib/kubelet/pods/e514b625-a309-4e35-b49f-7fbdadf82c0b/volumes" Nov 03 21:04:47.140747 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:47.140643 5815 scope.go:122] "RemoveContainer" containerID="f385928f5763b8c2870d02595f157af935f62884451cc9485d22d4d21b562657" Nov 03 21:04:47.292085 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:47.292043 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:04:48.026040 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:48.025998 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq"] Nov 03 21:04:48.030677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:48.030644 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db0ce8b6-bdksq"] Nov 03 21:04:48.585880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:48.585848 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbcdc2a-cc2c-46b4-84d5-e53a136fd066" path="/var/lib/kubelet/pods/7dbcdc2a-cc2c-46b4-84d5-e53a136fd066/volumes" Nov 03 21:04:55.432683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:55.432646 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz"] Nov 03 21:04:55.433782 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:55.433740 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" containerID="cri-o://16dc4aa6a187b3af9d7e99c1bc86a58367fe4235712268203764e64f19aa9b58" gracePeriod=30 Nov 03 21:04:56.343039 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:56.342996 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" podUID="e62cfde2-4364-452b-818c-43450e0ce767" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:04:59.038251 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:59.038192 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r"] Nov 03 21:04:59.041960 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:04:59.041931 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5069f49e-ncj9r"] Nov 03 21:05:00.246853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.246815 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6"] Nov 03 21:05:00.247379 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.247080 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" containerID="cri-o://7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab" gracePeriod=30 Nov 03 21:05:00.586646 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.585929 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deea3399-bc27-4898-9452-a4ae68c0be11" path="/var/lib/kubelet/pods/deea3399-bc27-4898-9452-a4ae68c0be11/volumes" Nov 03 21:05:00.586646 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.586406 5815 generic.go:356] "Generic (PLEG): container finished" podID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerID="16dc4aa6a187b3af9d7e99c1bc86a58367fe4235712268203764e64f19aa9b58" exitCode=0 Nov 03 21:05:00.586646 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.586600 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" event={"ID":"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a","Type":"ContainerDied","Data":"16dc4aa6a187b3af9d7e99c1bc86a58367fe4235712268203764e64f19aa9b58"} Nov 03 21:05:00.713067 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.713042 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 21:05:00.795799 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795766 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.795962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795831 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.795962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795862 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.795962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795884 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.795962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795905 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.795962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795925 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.796280 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.795997 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.796280 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796039 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.796280 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796080 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token\") pod \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\" (UID: \"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a\") " Nov 03 21:05:00.796280 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796125 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:00.796280 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796217 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:00.796549 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796337 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:00.796549 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796406 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:00.796659 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796620 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.796659 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796636 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.796659 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796645 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.796659 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796654 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.796798 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.796782 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:00.798993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.798966 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "kube-api-access-x8wqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:00.798993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.798976 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:00.799152 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.798998 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:00.799236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.799213 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token" pod "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" (UID: "ebd5c516-be9e-49f4-a2c5-4cb3e92af15a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:00.898013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.897934 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.898013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.897965 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.898013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.897976 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wqp\" (UniqueName: \"kubernetes.io/projected/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-kube-api-access-x8wqp\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.898013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.897986 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:00.898013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:00.897997 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.157197 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.157170 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 21:05:01.301153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.301121 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume\") pod \"e62cfde2-4364-452b-818c-43450e0ce767\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " Nov 03 21:05:01.301682 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.301228 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8\" (UniqueName: \"kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8\") pod \"e62cfde2-4364-452b-818c-43450e0ce767\" (UID: \"e62cfde2-4364-452b-818c-43450e0ce767\") " Nov 03 21:05:01.301682 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.301560 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume" pod "e62cfde2-4364-452b-818c-43450e0ce767" (UID: "e62cfde2-4364-452b-818c-43450e0ce767"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:01.303770 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.303742 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8" pod "e62cfde2-4364-452b-818c-43450e0ce767" (UID: "e62cfde2-4364-452b-818c-43450e0ce767"). InnerVolumeSpecName "kube-api-access-6qmp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:01.402150 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.402072 5815 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62cfde2-4364-452b-818c-43450e0ce767-plugins-config-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.402150 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.402099 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qmp8\" (UniqueName: \"kubernetes.io/projected/e62cfde2-4364-452b-818c-43450e0ce767-kube-api-access-6qmp8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.596191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.596156 5815 generic.go:356] "Generic (PLEG): container finished" podID="e62cfde2-4364-452b-818c-43450e0ce767" containerID="fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37" exitCode=0 Nov 03 21:05:01.596404 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.596203 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" Nov 03 21:05:01.596404 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.596240 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" event={"ID":"e62cfde2-4364-452b-818c-43450e0ce767","Type":"ContainerDied","Data":"fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37"} Nov 03 21:05:01.596404 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.596292 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r" event={"ID":"e62cfde2-4364-452b-818c-43450e0ce767","Type":"ContainerDied","Data":"5593134c025667399a039ade8e6a75cee98135025ca825cd176d512d1ffc4e1d"} Nov 03 21:05:01.596404 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.596322 5815 scope.go:122] "RemoveContainer" containerID="fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37" Nov 03 21:05:01.602814 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.602784 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" event={"ID":"ebd5c516-be9e-49f4-a2c5-4cb3e92af15a","Type":"ContainerDied","Data":"d2ea7d723e1f2dc164036e475ff77d025d67292bfe6d902b078ccfcdf05e5f33"} Nov 03 21:05:01.602814 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.602808 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz" Nov 03 21:05:01.613907 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.613891 5815 scope.go:122] "RemoveContainer" containerID="fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37" Nov 03 21:05:01.614172 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:05:01.614156 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37\": container with ID starting with fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37 not found: ID does not exist" containerID="fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37" Nov 03 21:05:01.614221 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.614181 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37"} err="failed to get container status \"fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37\": rpc error: code = NotFound desc = could not find container \"fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37\": container with ID starting with fdbee3e9fa09a915ca479c706affc3833f04242e28d20e6e7c6e53486eff6c37 not found: ID does not exist" Nov 03 21:05:01.614221 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.614197 5815 scope.go:122] "RemoveContainer" containerID="16dc4aa6a187b3af9d7e99c1bc86a58367fe4235712268203764e64f19aa9b58" Nov 03 21:05:01.630060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.630035 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r"] Nov 03 21:05:01.637537 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.633584 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-421604-7f8cfcd7c9-8fh4r"] Nov 03 21:05:01.657673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.657595 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz"] Nov 03 21:05:01.660826 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:01.660798 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-tbzkz"] Nov 03 21:05:02.585566 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:02.585531 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62cfde2-4364-452b-818c-43450e0ce767" path="/var/lib/kubelet/pods/e62cfde2-4364-452b-818c-43450e0ce767/volumes" Nov 03 21:05:02.586196 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:02.586176 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" path="/var/lib/kubelet/pods/ebd5c516-be9e-49f4-a2c5-4cb3e92af15a/volumes" Nov 03 21:05:02.895567 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:02.895481 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:05:05.514081 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.514027 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 21:05:05.633469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.633440 5815 generic.go:356] "Generic (PLEG): container finished" podID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerID="7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab" exitCode=0 Nov 03 21:05:05.633679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.633483 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" Nov 03 21:05:05.633679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.633545 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" event={"ID":"2f1f1ccc-1dca-4bbc-8eef-864382955a90","Type":"ContainerDied","Data":"7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab"} Nov 03 21:05:05.633679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.633597 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6" event={"ID":"2f1f1ccc-1dca-4bbc-8eef-864382955a90","Type":"ContainerDied","Data":"dffe0d3b7129e6ab42bf4763a507af7a603647e6d2790e9da614892c720c8cbc"} Nov 03 21:05:05.633679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.633621 5815 scope.go:122] "RemoveContainer" containerID="7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab" Nov 03 21:05:05.644460 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.644444 5815 scope.go:122] "RemoveContainer" containerID="7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab" Nov 03 21:05:05.644783 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:05:05.644760 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab\": container with ID starting with 7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab not found: ID does not exist" containerID="7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab" Nov 03 21:05:05.644841 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.644794 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab"} err="failed to get container status \"7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab\": rpc error: code = NotFound desc = could not find container \"7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab\": container with ID starting with 7035c83f50158883efff2b4e254484680dd0c7501655c021fdf6e006c3c6a3ab not found: ID does not exist" Nov 03 21:05:05.647088 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647068 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647111 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647131 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647272 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647252 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647309 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647296 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647384 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647368 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647451 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647413 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647525 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647469 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647525 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647414 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:05.647652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647498 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:05.647652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647423 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:05.647652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647518 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token\") pod \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\" (UID: \"2f1f1ccc-1dca-4bbc-8eef-864382955a90\") " Nov 03 21:05:05.647828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647656 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:05.647828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647754 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:05.648004 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.647980 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.648004 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.648006 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.648411 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.648016 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.648411 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.648025 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.648411 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.648033 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.649696 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.649675 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:05.649879 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.649863 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:05.650346 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.650321 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:05.650429 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.650342 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8" pod "2f1f1ccc-1dca-4bbc-8eef-864382955a90" (UID: "2f1f1ccc-1dca-4bbc-8eef-864382955a90"). InnerVolumeSpecName "kube-api-access-frnb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:05.749097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.749068 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.749097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.749093 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.749097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.749103 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f1ccc-1dca-4bbc-8eef-864382955a90-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.749313 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.749112 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frnb8\" (UniqueName: \"kubernetes.io/projected/2f1f1ccc-1dca-4bbc-8eef-864382955a90-kube-api-access-frnb8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:05.961205 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.961166 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6"] Nov 03 21:05:05.964739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:05.964713 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-vqnp6"] Nov 03 21:05:06.586888 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:06.586852 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" path="/var/lib/kubelet/pods/2f1f1ccc-1dca-4bbc-8eef-864382955a90/volumes" Nov 03 21:05:12.456518 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:12.456482 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:05:13.850999 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:05:13.850959 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:05:15.024274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:15.024240 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg"] Nov 03 21:05:15.026789 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:15.026765 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38df0deb-92xsg"] Nov 03 21:05:15.616138 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:15.615761 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp"] Nov 03 21:05:15.616138 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:15.616018 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" containerID="cri-o://be350a5510355906c146c510cde1a6d430bf500e4e0996d6319ffbcf04768dac" gracePeriod=30 Nov 03 21:05:16.483250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483218 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k"] Nov 03 21:05:16.483836 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483816 5815 cpu_manager.go:414] "containerMap: removing container" podUID="fa2a804a-a656-4bd8-803e-02007913f36d" containerName="payload-processing" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483839 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2a804a-a656-4bd8-803e-02007913f36d" containerName="payload-processing" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483859 5815 cpu_manager.go:414] "containerMap: removing container" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483867 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483896 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483904 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483927 5815 cpu_manager.go:414] "containerMap: removing container" podUID="341243bf-d4ba-447f-8317-326900c3535e" containerName="revoke-keys" Nov 03 21:05:16.483939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483934 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341243bf-d4ba-447f-8317-326900c3535e" containerName="revoke-keys" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483952 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e62cfde2-4364-452b-818c-43450e0ce767" containerName="payload-pre-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483960 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62cfde2-4364-452b-818c-43450e0ce767" containerName="payload-pre-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483979 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerName="payload-pre-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483988 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerName="payload-pre-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.483999 5815 cpu_manager.go:414] "containerMap: removing container" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484005 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484108 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62cfde2-4364-452b-818c-43450e0ce767" containerName="payload-pre-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484124 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e514b625-a309-4e35-b49f-7fbdadf82c0b" containerName="payload-pre-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484133 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="12da41d9-bf1f-4cd6-9993-ba7234f4c732" containerName="main" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484143 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2a804a-a656-4bd8-803e-02007913f36d" containerName="payload-processing" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484153 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1f1ccc-1dca-4bbc-8eef-864382955a90" containerName="istio-proxy" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484164 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd5c516-be9e-49f4-a2c5-4cb3e92af15a" containerName="istio-proxy" Nov 03 21:05:16.484335 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.484175 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="341243bf-d4ba-447f-8317-326900c3535e" containerName="revoke-keys" Nov 03 21:05:16.489124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.489104 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.494193 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.494169 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:05:16.494374 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.494358 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:05:16.496032 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.496005 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k"] Nov 03 21:05:16.585157 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.585125 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4001712-c784-4ec4-ba7e-37ba30678285" path="/var/lib/kubelet/pods/e4001712-c784-4ec4-ba7e-37ba30678285/volumes" Nov 03 21:05:16.610412 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.610382 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.610568 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.610436 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrl9\" (UniqueName: \"kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.610625 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.610607 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.711825 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.711788 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.712020 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.711848 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.712020 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.711911 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrl9\" (UniqueName: \"kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.712712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.712633 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.714616 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.714589 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.724252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.724230 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrl9\" (UniqueName: \"kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9\") pod \"maas-api-revoke-sub-7b51dbd5-xd99k\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:16.805444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:16.805380 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:17.088408 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:17.088381 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k"] Nov 03 21:05:17.089335 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:05:17.089308 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdd12fa_8a69_449a_a121_f7ef4261a218.slice/crio-d741e9e74ef77a8a0ee0f95fe6b154d092d7b4b5d22315fbfbc5ae29eedb3ed9 WatchSource:0}: Error finding container d741e9e74ef77a8a0ee0f95fe6b154d092d7b4b5d22315fbfbc5ae29eedb3ed9: Status 404 returned error can't find the container with id d741e9e74ef77a8a0ee0f95fe6b154d092d7b4b5d22315fbfbc5ae29eedb3ed9 Nov 03 21:05:17.723809 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:17.723777 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" event={"ID":"afdd12fa-8a69-449a-a121-f7ef4261a218","Type":"ContainerStarted","Data":"6d9ad55eb00e153e74be32789906628818bdcff13f8e7154d0e2c4e2b58b8c60"} Nov 03 21:05:17.724247 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:17.723835 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" event={"ID":"afdd12fa-8a69-449a-a121-f7ef4261a218","Type":"ContainerStarted","Data":"d741e9e74ef77a8a0ee0f95fe6b154d092d7b4b5d22315fbfbc5ae29eedb3ed9"} Nov 03 21:05:17.745877 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:17.745833 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" podStartSLOduration=1.7458189 podStartE2EDuration="1.7458189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:17.741688064 +0000 UTC m=+3827.272033736" watchObservedRunningTime="2026-11-03 21:05:17.7458189 +0000 UTC m=+3827.276164573" Nov 03 21:05:18.732447 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:18.732413 5815 generic.go:356] "Generic (PLEG): container finished" podID="afdd12fa-8a69-449a-a121-f7ef4261a218" containerID="6d9ad55eb00e153e74be32789906628818bdcff13f8e7154d0e2c4e2b58b8c60" exitCode=0 Nov 03 21:05:18.732883 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:18.732525 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" event={"ID":"afdd12fa-8a69-449a-a121-f7ef4261a218","Type":"ContainerDied","Data":"6d9ad55eb00e153e74be32789906628818bdcff13f8e7154d0e2c4e2b58b8c60"} Nov 03 21:05:19.878776 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:19.878685 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:20.047861 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.047774 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca\") pod \"afdd12fa-8a69-449a-a121-f7ef4261a218\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " Nov 03 21:05:20.047861 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.047835 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca\") pod \"afdd12fa-8a69-449a-a121-f7ef4261a218\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " Nov 03 21:05:20.047861 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.047860 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9\" (UniqueName: \"kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9\") pod \"afdd12fa-8a69-449a-a121-f7ef4261a218\" (UID: \"afdd12fa-8a69-449a-a121-f7ef4261a218\") " Nov 03 21:05:20.048230 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.048201 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca" pod "afdd12fa-8a69-449a-a121-f7ef4261a218" (UID: "afdd12fa-8a69-449a-a121-f7ef4261a218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:20.050237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.050211 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca" pod "afdd12fa-8a69-449a-a121-f7ef4261a218" (UID: "afdd12fa-8a69-449a-a121-f7ef4261a218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:20.050237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.050234 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9" pod "afdd12fa-8a69-449a-a121-f7ef4261a218" (UID: "afdd12fa-8a69-449a-a121-f7ef4261a218"). InnerVolumeSpecName "kube-api-access-dqrl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:20.148499 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.148469 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:20.148499 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.148496 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afdd12fa-8a69-449a-a121-f7ef4261a218-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:20.148687 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.148544 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrl9\" (UniqueName: \"kubernetes.io/projected/afdd12fa-8a69-449a-a121-f7ef4261a218-kube-api-access-dqrl9\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:20.748369 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.748336 5815 generic.go:356] "Generic (PLEG): container finished" podID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerID="be350a5510355906c146c510cde1a6d430bf500e4e0996d6319ffbcf04768dac" exitCode=0 Nov 03 21:05:20.748497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.748406 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" event={"ID":"b5ba818e-adb1-4432-ac0a-ca912d3c28c1","Type":"ContainerDied","Data":"be350a5510355906c146c510cde1a6d430bf500e4e0996d6319ffbcf04768dac"} Nov 03 21:05:20.748497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.748477 5815 scope.go:122] "RemoveContainer" containerID="97d792a1ff7646d96b29370c2caf519086eee8a8794b18c64b119a79262dcc73" Nov 03 21:05:20.750245 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.750228 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" Nov 03 21:05:20.750365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.750254 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k" event={"ID":"afdd12fa-8a69-449a-a121-f7ef4261a218","Type":"ContainerDied","Data":"d741e9e74ef77a8a0ee0f95fe6b154d092d7b4b5d22315fbfbc5ae29eedb3ed9"} Nov 03 21:05:20.750365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.750300 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d741e9e74ef77a8a0ee0f95fe6b154d092d7b4b5d22315fbfbc5ae29eedb3ed9" Nov 03 21:05:20.887158 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:20.887133 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 21:05:21.055920 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.055887 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056128 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.055932 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056128 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.055977 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056128 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056004 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056128 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056099 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056130 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056213 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056260 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056292 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data\") pod \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\" (UID: \"b5ba818e-adb1-4432-ac0a-ca912d3c28c1\") " Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056331 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056353 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:21.056383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056375 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:21.056830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056620 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:21.056830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056688 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:21.056830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056744 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.056830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056762 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.056830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056777 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.056830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.056804 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.058392 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.058368 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:21.058528 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.058471 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:21.058925 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.058905 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:21.059006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.058916 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs" pod "b5ba818e-adb1-4432-ac0a-ca912d3c28c1" (UID: "b5ba818e-adb1-4432-ac0a-ca912d3c28c1"). InnerVolumeSpecName "kube-api-access-qs6gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:21.157564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.157462 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.157564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.157490 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.157564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.157501 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6gs\" (UniqueName: \"kubernetes.io/projected/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-kube-api-access-qs6gs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.157564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.157527 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.157564 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.157542 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ba818e-adb1-4432-ac0a-ca912d3c28c1-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.759994 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.759957 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" Nov 03 21:05:21.760184 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.759994 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp" event={"ID":"b5ba818e-adb1-4432-ac0a-ca912d3c28c1","Type":"ContainerDied","Data":"4392213ec9cc5f9bb8ee45fd9534674e9a5a931e6200fb3a86189aaac5d022e8"} Nov 03 21:05:21.760184 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.760056 5815 scope.go:122] "RemoveContainer" containerID="be350a5510355906c146c510cde1a6d430bf500e4e0996d6319ffbcf04768dac" Nov 03 21:05:21.784569 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.784538 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp"] Nov 03 21:05:21.791233 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:21.791209 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-bw8tp"] Nov 03 21:05:22.584054 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:22.584011 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" path="/var/lib/kubelet/pods/b5ba818e-adb1-4432-ac0a-ca912d3c28c1/volumes" Nov 03 21:05:23.066257 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:23.066217 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:05:26.023584 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:26.023549 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x"] Nov 03 21:05:26.026104 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:26.026082 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be52a41e-q5w9x"] Nov 03 21:05:26.584184 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:26.584152 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c009781-2002-4e35-b3d6-ea9f3c023008" path="/var/lib/kubelet/pods/5c009781-2002-4e35-b3d6-ea9f3c023008/volumes" Nov 03 21:05:28.475222 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.475190 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69"] Nov 03 21:05:28.475817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.475797 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" Nov 03 21:05:28.475904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.475819 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" Nov 03 21:05:28.475904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.475847 5815 cpu_manager.go:414] "containerMap: removing container" podUID="afdd12fa-8a69-449a-a121-f7ef4261a218" containerName="revoke-keys" Nov 03 21:05:28.475904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.475856 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdd12fa-8a69-449a-a121-f7ef4261a218" containerName="revoke-keys" Nov 03 21:05:28.476067 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.475989 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" Nov 03 21:05:28.476067 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.476004 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdd12fa-8a69-449a-a121-f7ef4261a218" containerName="revoke-keys" Nov 03 21:05:28.476067 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.476019 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" Nov 03 21:05:28.480845 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.480822 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.485525 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.485485 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:05:28.485709 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.485589 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:05:28.488819 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.488788 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69"] Nov 03 21:05:28.637837 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.637761 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.637837 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.637814 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.638037 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.637936 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmht\" (UniqueName: \"kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.739357 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.739322 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.739559 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.739378 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.739559 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.739429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmht\" (UniqueName: \"kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.739986 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.739957 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.742048 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.742026 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.752105 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.752083 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmht\" (UniqueName: \"kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht\") pod \"maas-api-revoke-sub-402ae974-rbr69\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:28.795956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:28.795929 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:29.080832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:29.080803 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69"] Nov 03 21:05:29.082412 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:05:29.082381 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa41ed5_88cc_41be_841d_3628e8bba551.slice/crio-3796c6cd81b55c210deb9db0e3c736e8384133b6d545387bcd39062c3d4e8934 WatchSource:0}: Error finding container 3796c6cd81b55c210deb9db0e3c736e8384133b6d545387bcd39062c3d4e8934: Status 404 returned error can't find the container with id 3796c6cd81b55c210deb9db0e3c736e8384133b6d545387bcd39062c3d4e8934 Nov 03 21:05:29.824761 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:29.824723 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" event={"ID":"8aa41ed5-88cc-41be-841d-3628e8bba551","Type":"ContainerStarted","Data":"22c73fa351ce06cb15c242b69b86c0c9c81bcb5cd3b1da235c4e6beeabfd070b"} Nov 03 21:05:29.825185 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:29.824785 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" event={"ID":"8aa41ed5-88cc-41be-841d-3628e8bba551","Type":"ContainerStarted","Data":"3796c6cd81b55c210deb9db0e3c736e8384133b6d545387bcd39062c3d4e8934"} Nov 03 21:05:29.843024 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:29.842980 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" podStartSLOduration=1.8429608 podStartE2EDuration="1.8429608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:29.84204804 +0000 UTC m=+3839.372393713" watchObservedRunningTime="2026-11-03 21:05:29.8429608 +0000 UTC m=+3839.373306475" Nov 03 21:05:30.833118 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:30.833085 5815 generic.go:356] "Generic (PLEG): container finished" podID="8aa41ed5-88cc-41be-841d-3628e8bba551" containerID="22c73fa351ce06cb15c242b69b86c0c9c81bcb5cd3b1da235c4e6beeabfd070b" exitCode=0 Nov 03 21:05:30.833578 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:30.833163 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" event={"ID":"8aa41ed5-88cc-41be-841d-3628e8bba551","Type":"ContainerDied","Data":"22c73fa351ce06cb15c242b69b86c0c9c81bcb5cd3b1da235c4e6beeabfd070b"} Nov 03 21:05:31.986375 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:31.986349 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:32.066544 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.066488 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca\") pod \"8aa41ed5-88cc-41be-841d-3628e8bba551\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " Nov 03 21:05:32.066752 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.066629 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca\") pod \"8aa41ed5-88cc-41be-841d-3628e8bba551\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " Nov 03 21:05:32.066752 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.066688 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht\" (UniqueName: \"kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht\") pod \"8aa41ed5-88cc-41be-841d-3628e8bba551\" (UID: \"8aa41ed5-88cc-41be-841d-3628e8bba551\") " Nov 03 21:05:32.066951 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.066928 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca" pod "8aa41ed5-88cc-41be-841d-3628e8bba551" (UID: "8aa41ed5-88cc-41be-841d-3628e8bba551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:32.068866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.068842 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca" pod "8aa41ed5-88cc-41be-841d-3628e8bba551" (UID: "8aa41ed5-88cc-41be-841d-3628e8bba551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:32.068946 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.068886 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht" pod "8aa41ed5-88cc-41be-841d-3628e8bba551" (UID: "8aa41ed5-88cc-41be-841d-3628e8bba551"). InnerVolumeSpecName "kube-api-access-nqmht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:32.168392 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.168327 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.168392 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.168353 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmht\" (UniqueName: \"kubernetes.io/projected/8aa41ed5-88cc-41be-841d-3628e8bba551-kube-api-access-nqmht\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.168392 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.168363 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa41ed5-88cc-41be-841d-3628e8bba551-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:32.848777 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.848690 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" event={"ID":"8aa41ed5-88cc-41be-841d-3628e8bba551","Type":"ContainerDied","Data":"3796c6cd81b55c210deb9db0e3c736e8384133b6d545387bcd39062c3d4e8934"} Nov 03 21:05:32.848777 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.848719 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69" Nov 03 21:05:32.848777 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:32.848739 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3796c6cd81b55c210deb9db0e3c736e8384133b6d545387bcd39062c3d4e8934" Nov 03 21:05:41.245942 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:41.245904 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:05:43.849573 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:05:43.849536 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:05:47.244581 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:05:47.244551 5815 scope.go:122] "RemoveContainer" containerID="83d7ee4fc9cd90f83dc157a7f803dd6c9faa8529cd88744512b52c518bf8b228" Nov 03 21:06:00.960378 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:00.960338 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc"] Nov 03 21:06:00.960973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:00.960627 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" containerID="cri-o://692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473" gracePeriod=30 Nov 03 21:06:02.185612 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:02.185550 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-bx9l8_2276a55f-e1a9-406a-994e-ace6113b9d42/manager/0.log" Nov 03 21:06:02.369248 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:02.369218 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jgwq4_df4b5345-2b3d-425d-a9f9-2dc3f382fcf7/manager/0.log" Nov 03 21:06:02.968078 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:02.968039 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-k4psh_e79d3bb8-3533-4def-a7d4-2fba2b589a0f/manager/0.log" Nov 03 21:06:03.210957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:03.210927 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-q9gdw_683dd503-5b16-4acc-9d0f-36fd1baf048e/limitador/0.log" Nov 03 21:06:03.399770 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:03.399698 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h946s_9da9ab34-02dc-458e-84d7-e9522d44c516/manager/0.log" Nov 03 21:06:04.927860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:04.927825 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q5w7x_09dae28a-2531-444a-bf0c-bbe040b3b92d/manager/0.log" Nov 03 21:06:06.215342 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.215314 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 21:06:06.381385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381299 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381385 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381358 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381428 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381475 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381535 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381569 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381599 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.381683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381659 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.382019 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381696 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket\") pod \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\" (UID: \"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff\") " Nov 03 21:06:06.382019 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381835 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.382019 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381861 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:06.382019 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.381886 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.382252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.382133 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.382252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.382151 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.382252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.382171 5815 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.382252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.382196 5815 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-data\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.382677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.382216 5815 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istiod-ca-cert\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.383784 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.383756 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:06.383936 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.383901 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:06.383936 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.383915 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.384097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.384079 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6" pod "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" (UID: "db20e08e-a4c7-4bf7-b9de-f79ad70e23ff"). InnerVolumeSpecName "kube-api-access-2c7s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:06.482977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.482949 5815 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-token\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.482977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.482974 5815 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-envoy\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.483153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.482986 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7s6\" (UniqueName: \"kubernetes.io/projected/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-kube-api-access-2c7s6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.483153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.482996 5815 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-workload-certs\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.483153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.483005 5815 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-credential-socket\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.483153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:06.483013 5815 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff-istio-podinfo\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:07.076402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.076365 5815 generic.go:356] "Generic (PLEG): container finished" podID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerID="692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473" exitCode=0 Nov 03 21:06:07.076608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.076420 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" Nov 03 21:06:07.076608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.076450 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" event={"ID":"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff","Type":"ContainerDied","Data":"692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473"} Nov 03 21:06:07.076608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.076527 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc" event={"ID":"db20e08e-a4c7-4bf7-b9de-f79ad70e23ff","Type":"ContainerDied","Data":"815b7b099c99ab1e3c03ce0012dad9ca68d145e92028c2b1ae7f96b502e205fd"} Nov 03 21:06:07.076608 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.076557 5815 scope.go:122] "RemoveContainer" containerID="692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473" Nov 03 21:06:07.087236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.086995 5815 scope.go:122] "RemoveContainer" containerID="692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473" Nov 03 21:06:07.087330 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:06:07.087312 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473\": container with ID starting with 692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473 not found: ID does not exist" containerID="692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473" Nov 03 21:06:07.087377 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.087340 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473"} err="failed to get container status \"692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473\": rpc error: code = NotFound desc = could not find container \"692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473\": container with ID starting with 692c9d642905365ee99aa75c21b73d21df24db18314a5e4a3774924165fd0473 not found: ID does not exist" Nov 03 21:06:07.097730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.097701 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc"] Nov 03 21:06:07.102249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:07.102228 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wcctc"] Nov 03 21:06:08.583982 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:08.583909 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" path="/var/lib/kubelet/pods/db20e08e-a4c7-4bf7-b9de-f79ad70e23ff/volumes" Nov 03 21:06:13.853165 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:06:13.853123 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:06:16.944128 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:16.944090 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:06:24.033099 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:24.033065 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k"] Nov 03 21:06:24.035725 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:24.035701 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b19140-tlj5k"] Nov 03 21:06:24.583726 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:24.583690 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ab6af4-cfb9-4054-a616-174cc898d9b0" path="/var/lib/kubelet/pods/a0ab6af4-cfb9-4054-a616-174cc898d9b0/volumes" Nov 03 21:06:35.023869 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:35.023831 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq"] Nov 03 21:06:35.026380 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:35.026353 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3582fe58-xlxcq"] Nov 03 21:06:36.271268 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:36.271240 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:06:36.277324 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:36.277302 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:06:36.295921 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:36.295902 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:06:36.301291 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:36.301275 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:06:36.583997 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:36.583919 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25adebeb-57a1-4c4f-9f92-a594dce7c388" path="/var/lib/kubelet/pods/25adebeb-57a1-4c4f-9f92-a594dce7c388/volumes" Nov 03 21:06:42.152248 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:42.152208 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:06:43.851555 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:06:43.851493 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:06:45.033313 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:45.033278 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l"] Nov 03 21:06:45.037779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:45.037740 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495a507d-gwf4l"] Nov 03 21:06:46.584489 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:46.584451 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341dda15-c4d4-43cf-baef-bdc4a3cfc6d1" path="/var/lib/kubelet/pods/341dda15-c4d4-43cf-baef-bdc4a3cfc6d1/volumes" Nov 03 21:06:47.287560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:47.287528 5815 scope.go:122] "RemoveContainer" containerID="5c0c8c1bcdb623f1798937d9a08df366e3d917e3e8bfc182ce3f34b5748b412f" Nov 03 21:06:47.297180 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:47.297159 5815 scope.go:122] "RemoveContainer" containerID="20d8952a444b51e8ece710e2605765cd17263c2bfc9bd25fba64acf265ebced9" Nov 03 21:06:47.305963 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:47.305939 5815 scope.go:122] "RemoveContainer" containerID="d45dfb7a7bd4d6673ef60630efb37e59adc54df5bbb2923d5f734c467c976d47" Nov 03 21:06:55.033653 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:55.033617 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v"] Nov 03 21:06:55.036262 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:55.036238 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31489283-hgq7v"] Nov 03 21:06:56.584279 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:06:56.584237 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51770a0b-05ca-424f-8bbd-0cda586850a4" path="/var/lib/kubelet/pods/51770a0b-05ca-424f-8bbd-0cda586850a4/volumes" Nov 03 21:07:03.339242 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339208 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm"] Nov 03 21:07:03.339685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339633 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8aa41ed5-88cc-41be-841d-3628e8bba551" containerName="revoke-keys" Nov 03 21:07:03.339685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339647 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa41ed5-88cc-41be-841d-3628e8bba551" containerName="revoke-keys" Nov 03 21:07:03.339685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339658 5815 cpu_manager.go:414] "containerMap: removing container" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" Nov 03 21:07:03.339685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339663 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ba818e-adb1-4432-ac0a-ca912d3c28c1" containerName="istio-proxy" Nov 03 21:07:03.339685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339678 5815 cpu_manager.go:414] "containerMap: removing container" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" Nov 03 21:07:03.339685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339684 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" Nov 03 21:07:03.339898 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339766 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa41ed5-88cc-41be-841d-3628e8bba551" containerName="revoke-keys" Nov 03 21:07:03.339898 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.339776 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="db20e08e-a4c7-4bf7-b9de-f79ad70e23ff" containerName="istio-proxy" Nov 03 21:07:03.344663 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.344640 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.372464 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.372429 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm"] Nov 03 21:07:03.381408 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.381381 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb"] Nov 03 21:07:03.385269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.385250 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.397389 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.397351 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb"] Nov 03 21:07:03.461186 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461159 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461194 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44953e3d-f118-443e-894d-530fba487260-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461210 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461302 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdsnl\" (UniqueName: \"kubernetes.io/projected/da9aeac9-c831-4be9-9aba-39a42918b4f6-kube-api-access-mdsnl\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461540 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461354 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7sf\" (UniqueName: \"kubernetes.io/projected/44953e3d-f118-443e-894d-530fba487260-kube-api-access-jx7sf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461540 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461410 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461540 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461445 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461540 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461473 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9aeac9-c831-4be9-9aba-39a42918b4f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461540 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461488 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461749 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461582 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461749 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461613 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461749 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461641 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461749 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461670 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461749 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461706 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461749 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461737 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461981 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461767 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44953e3d-f118-443e-894d-530fba487260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.461981 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461840 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.461981 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.461858 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44953e3d-f118-443e-894d-530fba487260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.563122 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563096 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563132 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44953e3d-f118-443e-894d-530fba487260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.563295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563152 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563167 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44953e3d-f118-443e-894d-530fba487260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.563295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563207 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.563589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563386 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44953e3d-f118-443e-894d-530fba487260-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.563589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563419 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563457 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdsnl\" (UniqueName: \"kubernetes.io/projected/da9aeac9-c831-4be9-9aba-39a42918b4f6-kube-api-access-mdsnl\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563483 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7sf\" (UniqueName: \"kubernetes.io/projected/44953e3d-f118-443e-894d-530fba487260-kube-api-access-jx7sf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.563589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563535 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563565 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563593 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9aeac9-c831-4be9-9aba-39a42918b4f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563603 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563616 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.563956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563671 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.564290 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564265 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.564468 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563913 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.564645 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564040 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44953e3d-f118-443e-894d-530fba487260-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564306 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564682 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564716 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564354 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564527 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da9aeac9-c831-4be9-9aba-39a42918b4f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.563774 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.564779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564775 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.565237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.564802 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.565237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.565131 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.565237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.565231 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.566090 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.566065 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44953e3d-f118-443e-894d-530fba487260-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.566913 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.566890 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.567061 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.567036 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44953e3d-f118-443e-894d-530fba487260-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.567174 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.567155 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.575600 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.575579 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44953e3d-f118-443e-894d-530fba487260-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.575738 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.575683 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da9aeac9-c831-4be9-9aba-39a42918b4f6-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.576391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.576372 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdsnl\" (UniqueName: \"kubernetes.io/projected/da9aeac9-c831-4be9-9aba-39a42918b4f6-kube-api-access-mdsnl\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xvscb\" (UID: \"da9aeac9-c831-4be9-9aba-39a42918b4f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.576487 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.576411 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7sf\" (UniqueName: \"kubernetes.io/projected/44953e3d-f118-443e-894d-530fba487260-kube-api-access-jx7sf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-h8zwm\" (UID: \"44953e3d-f118-443e-894d-530fba487260\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.671595 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.671535 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:03.700364 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.700337 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:03.958077 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.958048 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm"] Nov 03 21:07:03.958616 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:07:03.958593 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44953e3d_f118_443e_894d_530fba487260.slice/crio-e0ea108756993fe23f4260bff1ebb6167c0a2cb5e8518904ad5d8218a3710cd7 WatchSource:0}: Error finding container e0ea108756993fe23f4260bff1ebb6167c0a2cb5e8518904ad5d8218a3710cd7: Status 404 returned error can't find the container with id e0ea108756993fe23f4260bff1ebb6167c0a2cb5e8518904ad5d8218a3710cd7 Nov 03 21:07:03.960774 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.960744 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:07:03.960844 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.960807 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:07:03.960844 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.960835 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:07:03.997152 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:03.997129 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb"] Nov 03 21:07:03.999237 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:07:03.999210 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9aeac9_c831_4be9_9aba_39a42918b4f6.slice/crio-1727f95a6391a772bf0d32684dfb8cfdb2b58c7e137638f0f7cb5d11fd57f468 WatchSource:0}: Error finding container 1727f95a6391a772bf0d32684dfb8cfdb2b58c7e137638f0f7cb5d11fd57f468: Status 404 returned error can't find the container with id 1727f95a6391a772bf0d32684dfb8cfdb2b58c7e137638f0f7cb5d11fd57f468 Nov 03 21:07:04.001577 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.001533 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:07:04.001688 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.001604 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:07:04.001688 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.001632 5815 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 21:07:04.452727 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.452685 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" event={"ID":"da9aeac9-c831-4be9-9aba-39a42918b4f6","Type":"ContainerStarted","Data":"b48c55f669d00f0170dde81ec2b4bcf32c1284aac5ba5af6d965f2a33f9fc083"} Nov 03 21:07:04.453269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.452747 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" event={"ID":"da9aeac9-c831-4be9-9aba-39a42918b4f6","Type":"ContainerStarted","Data":"1727f95a6391a772bf0d32684dfb8cfdb2b58c7e137638f0f7cb5d11fd57f468"} Nov 03 21:07:04.454902 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.454861 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" event={"ID":"44953e3d-f118-443e-894d-530fba487260","Type":"ContainerStarted","Data":"90b8975ec2c24933da11f810d635380ef7a72c2a76adf21a91f85ca03ad1091a"} Nov 03 21:07:04.455048 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.454919 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" event={"ID":"44953e3d-f118-443e-894d-530fba487260","Type":"ContainerStarted","Data":"e0ea108756993fe23f4260bff1ebb6167c0a2cb5e8518904ad5d8218a3710cd7"} Nov 03 21:07:04.520531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.515923 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podStartSLOduration=1.515901263 podStartE2EDuration="1.515901263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:04.484830008 +0000 UTC m=+3934.015175938" watchObservedRunningTime="2026-11-03 21:07:04.515901263 +0000 UTC m=+3934.046246937" Nov 03 21:07:04.676532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.673610 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:04.701700 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:04.701608 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:05.680651 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:05.680613 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:05.681539 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:05.681306 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:05.703867 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:05.703835 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:05.704103 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:05.704075 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:06.029984 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.029927 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podStartSLOduration=3.029911379 podStartE2EDuration="3.029911379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:04.521919454 +0000 UTC m=+3934.052265127" watchObservedRunningTime="2026-11-03 21:07:06.029911379 +0000 UTC m=+3935.560257056" Nov 03 21:07:06.032031 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.032007 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx"] Nov 03 21:07:06.034718 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.034697 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9e5cdca-bcshx"] Nov 03 21:07:06.614120 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.614076 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077a476d-da55-4fd8-8f91-62d84f5a4d34" path="/var/lib/kubelet/pods/077a476d-da55-4fd8-8f91-62d84f5a4d34/volumes" Nov 03 21:07:06.688017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.687983 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:06.688627 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.688594 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:06.714702 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.714602 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:06.714702 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:06.714649 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:07.677882 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:07.677845 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:07.678174 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:07.678144 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:07.700856 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:07.700826 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:07.701397 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:07.701371 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:08.683535 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:08.672974 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:08.683535 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:08.673038 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:08.703189 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:08.703156 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:08.704220 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:08.703805 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:09.677924 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:09.677885 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:09.678229 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:09.678200 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:09.704250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:09.704217 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:09.704880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:09.704844 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:10.055224 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.055187 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf"] Nov 03 21:07:10.058391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.058362 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abd08420-cdzjf"] Nov 03 21:07:10.586237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.586184 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e43a07b-30d8-42ef-ae78-36a2f37ea6d1" path="/var/lib/kubelet/pods/8e43a07b-30d8-42ef-ae78-36a2f37ea6d1/volumes" Nov 03 21:07:10.672823 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.672786 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:10.673004 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.672851 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:10.701023 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.700987 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:10.701173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:10.701044 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:11.672244 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:11.672209 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" start-of-body= Nov 03 21:07:11.672707 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:11.672272 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": dial tcp 10.132.1.205:15021: connect: connection refused" Nov 03 21:07:11.701684 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:11.701652 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" start-of-body= Nov 03 21:07:11.701822 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:11.701706 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": dial tcp 10.132.1.206:15021: connect: connection refused" Nov 03 21:07:13.672953 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:13.672908 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:13.673641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:13.673602 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:13.701790 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:13.701758 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:13.701951 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:13.701813 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:13.852069 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:07:13.852028 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:07:14.714681 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:14.714635 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:14.715217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:14.714696 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:14.742669 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:14.742640 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:14.742907 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:14.742868 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:15.755679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.755628 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:15.756252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.755701 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:15.785643 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.785614 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:15.785822 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.785664 5815 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:15.790406 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.790381 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:15.790977 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.790952 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:15.791613 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.791591 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" Nov 03 21:07:15.797285 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.797265 5815 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:15.797560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.797545 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:15.798268 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:15.798254 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" Nov 03 21:07:18.025339 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:18.025298 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2"] Nov 03 21:07:18.027975 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:18.027949 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a73c9e9-5kqc2"] Nov 03 21:07:18.584223 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:18.584191 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29721775-b0a9-4845-9c2b-fe0d5c67d11e" path="/var/lib/kubelet/pods/29721775-b0a9-4845-9c2b-fe0d5c67d11e/volumes" Nov 03 21:07:20.026172 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:20.026138 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz"] Nov 03 21:07:20.028456 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:20.028436 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24787d73-vs5vz"] Nov 03 21:07:20.586049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:20.586016 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d6a935-afc8-4ae0-8b73-9c6d4a0c314a" path="/var/lib/kubelet/pods/94d6a935-afc8-4ae0-8b73-9c6d4a0c314a/volumes" Nov 03 21:07:43.852812 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:07:43.852712 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:07:45.777029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:45.776993 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:07:47.346533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:47.346116 5815 scope.go:122] "RemoveContainer" containerID="34e6adcb43a00252bab794797a7d18e3ab8de9a2b541f4b73ad1b5ec7303f4b7" Nov 03 21:07:47.361660 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:47.361628 5815 scope.go:122] "RemoveContainer" containerID="8133df8cd11b2c55cdebd14b88cc654dc4a163578e1475940ffc3a261dee839e" Nov 03 21:07:47.419735 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:07:47.417612 5815 scope.go:122] "RemoveContainer" containerID="6ab2b647702b654d2090d9ba37661c2fac31481ad5a8bea53b1af0242dfe7030" Nov 03 21:08:01.756962 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:01.756924 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:08:07.077563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:07.077530 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:08:12.307185 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:12.307141 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:08:13.851469 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:08:13.851433 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:08:25.034319 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:25.034277 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8"] Nov 03 21:08:25.037169 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:25.037019 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-864d592c-r7jd8"] Nov 03 21:08:25.654762 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:25.654724 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:08:26.584407 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:26.584371 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578097e1-6482-4606-bcb4-8dd74dca55eb" path="/var/lib/kubelet/pods/578097e1-6482-4606-bcb4-8dd74dca55eb/volumes" Nov 03 21:08:32.028700 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:32.028659 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c"] Nov 03 21:08:32.029215 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:32.028717 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696db94b-6hb9c"] Nov 03 21:08:32.583975 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:32.583932 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73336eb9-eaea-48b1-b58e-361c5d908aa1" path="/var/lib/kubelet/pods/73336eb9-eaea-48b1-b58e-361c5d908aa1/volumes" Nov 03 21:08:36.022873 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:36.022838 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv"] Nov 03 21:08:36.025765 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:36.025728 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05cbe9f-7wbqv"] Nov 03 21:08:36.584558 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:36.584500 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eccc426-4b20-485d-8a90-f6e7174f4788" path="/var/lib/kubelet/pods/5eccc426-4b20-485d-8a90-f6e7174f4788/volumes" Nov 03 21:08:42.021574 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:42.021541 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs"] Nov 03 21:08:42.026146 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:42.026121 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b457452-qp9xs"] Nov 03 21:08:42.584568 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:42.584534 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed13b7f1-a0d2-4ff8-b130-72069b89b501" path="/var/lib/kubelet/pods/ed13b7f1-a0d2-4ff8-b130-72069b89b501/volumes" Nov 03 21:08:43.851680 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:08:43.851646 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:08:47.565274 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:47.565245 5815 scope.go:122] "RemoveContainer" containerID="0e0cb0a4e57d6e58ec02171f6eed1a0944fb4d6eedb523e86fa2f3991465ddeb" Nov 03 21:08:47.577882 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:47.577862 5815 scope.go:122] "RemoveContainer" containerID="5407caaa16bd2181381df64bbee9db3431c894ef87883a41eee35023d15f487e" Nov 03 21:08:47.586919 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:47.586893 5815 scope.go:122] "RemoveContainer" containerID="df2eccbb4a1a855f2606694eb3eb7e2f33ec60b1a9d9fdaa9b42b42e270d1d82" Nov 03 21:08:47.595448 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:47.595418 5815 scope.go:122] "RemoveContainer" containerID="b48299085189931ac8b539cc8de8146cc10b8149a03e9c54dd14afb04f1e9cad" Nov 03 21:08:47.603866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:47.603848 5815 scope.go:122] "RemoveContainer" containerID="23fe75d42ba9e08b34c6bf00495bdf2088f399b81424d682879b78af97dae2cd" Nov 03 21:08:53.033500 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:53.033460 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6"] Nov 03 21:08:53.035994 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:53.035961 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-839954d0-gsdf6"] Nov 03 21:08:53.331028 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:53.330949 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:08:54.600197 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:54.599749 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0db79b-4963-47ec-897b-d3248f62d373" path="/var/lib/kubelet/pods/3c0db79b-4963-47ec-897b-d3248f62d373/volumes" Nov 03 21:08:59.954934 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:08:59.954891 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:09:04.022806 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:04.022768 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr"] Nov 03 21:09:04.025568 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:04.025544 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45fb1561-8hdmr"] Nov 03 21:09:04.583656 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:04.583616 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e903e189-38be-4525-a3f2-7cbf8a6f8503" path="/var/lib/kubelet/pods/e903e189-38be-4525-a3f2-7cbf8a6f8503/volumes" Nov 03 21:09:11.631333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:11.631252 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:09:13.959523 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:09:13.959426 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:09:15.035532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:15.032271 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88"] Nov 03 21:09:15.036164 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:15.036144 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23e031ee-s6z88"] Nov 03 21:09:16.585320 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:16.585285 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa2e081-bd00-492f-b3f7-8c0d537f5744" path="/var/lib/kubelet/pods/aaa2e081-bd00-492f-b3f7-8c0d537f5744/volumes" Nov 03 21:09:17.184346 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:17.184312 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:09:33.022747 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:33.022709 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp"] Nov 03 21:09:33.025225 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:33.025202 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b8ba30-ghbvp"] Nov 03 21:09:34.584226 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:34.584193 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341243bf-d4ba-447f-8317-326900c3535e" path="/var/lib/kubelet/pods/341243bf-d4ba-447f-8317-326900c3535e/volumes" Nov 03 21:09:43.850337 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:09:43.850299 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:09:47.643003 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:47.642972 5815 scope.go:122] "RemoveContainer" containerID="d8d5aade4f7649d59f6a677d75bbc7081421eea8596ba6dd989a1eca75999fb4" Nov 03 21:09:47.652370 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:47.652353 5815 scope.go:122] "RemoveContainer" containerID="48dd80457aeb832e9b0d939b106be4a7f7939fee3da961ac42496e67e24d92c5" Nov 03 21:09:47.660896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:47.660882 5815 scope.go:122] "RemoveContainer" containerID="5004bb90068d47dd7f75ed9231cdf7aaf2868047c866a5f69cc10deed152cc23" Nov 03 21:09:47.668946 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:47.668923 5815 scope.go:122] "RemoveContainer" containerID="18601fea8e5c80a308a14f6c56f4308ac9a63b045891ad0a99ce6d0d33ca1594" Nov 03 21:09:56.062037 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:09:56.061999 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:10:13.850316 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:10:13.850281 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:10:15.471557 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:15.471498 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:10:20.034759 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:20.034723 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k"] Nov 03 21:10:20.035384 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:20.035367 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b51dbd5-xd99k"] Nov 03 21:10:20.256137 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:20.256104 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:10:20.584915 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:20.584886 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdd12fa-8a69-449a-a121-f7ef4261a218" path="/var/lib/kubelet/pods/afdd12fa-8a69-449a-a121-f7ef4261a218/volumes" Nov 03 21:10:32.024908 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:32.024871 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69"] Nov 03 21:10:32.027905 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:32.027856 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-402ae974-rbr69"] Nov 03 21:10:32.584493 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:32.584460 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa41ed5-88cc-41be-841d-3628e8bba551" path="/var/lib/kubelet/pods/8aa41ed5-88cc-41be-841d-3628e8bba551/volumes" Nov 03 21:10:39.857791 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:39.857718 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:10:43.850317 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:10:43.850280 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:10:47.707551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:47.707521 5815 scope.go:122] "RemoveContainer" containerID="3dbabf118d8d04e34e8686f57d506e6675517bda934b926ff8276eacb6720e89" Nov 03 21:10:47.716605 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:47.716586 5815 scope.go:122] "RemoveContainer" containerID="dbe9b688bf3960b74433f2355fe4627f94a38ace67d934de10c259dcdacd8da1" Nov 03 21:10:47.744378 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:47.744349 5815 scope.go:122] "RemoveContainer" containerID="c03a982609098ab2b1e99fd368f7aa41d9ef3696b75dd4533513e5aaa58f8c71" Nov 03 21:10:47.755925 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:47.755904 5815 scope.go:122] "RemoveContainer" containerID="3c14af120fc3b1f8af361eccd467f9d9a7a2e22a16c68d3f82f80b90448c23a3" Nov 03 21:10:54.488453 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:10:54.488419 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:11:11.682732 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:11.682688 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:11:13.860995 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:11:13.860953 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:11:23.065309 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.065272 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6"] Nov 03 21:11:23.068682 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.068666 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.077298 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.077271 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:11:23.077466 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.077392 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:11:23.078252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.078228 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6"] Nov 03 21:11:23.240563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.240527 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx6fw\" (UniqueName: \"kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.240769 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.240582 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.240769 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.240741 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.342251 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.342154 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx6fw\" (UniqueName: \"kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.342251 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.342210 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.342556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.342328 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.343032 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.343005 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.345029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.345000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.354735 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.354711 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx6fw\" (UniqueName: \"kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw\") pod \"maas-api-revoke-sub-8c7f157a-6zlp6\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.383912 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.383888 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:23.662124 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:23.662100 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6"] Nov 03 21:11:23.664327 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:11:23.664302 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76ed6c9_e074_4145_9235_147add5f25b1.slice/crio-aa32b4ad15a7481e3b9e45901e21412e4cb48dabfe68ba27e3592dac78379804 WatchSource:0}: Error finding container aa32b4ad15a7481e3b9e45901e21412e4cb48dabfe68ba27e3592dac78379804: Status 404 returned error can't find the container with id aa32b4ad15a7481e3b9e45901e21412e4cb48dabfe68ba27e3592dac78379804 Nov 03 21:11:24.422015 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:24.421935 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" event={"ID":"e76ed6c9-e074-4145-9235-147add5f25b1","Type":"ContainerStarted","Data":"a60630a3c422920d71db8f3efaad72d47ae13c11da3c48cd8a5adfcd9bfaa26e"} Nov 03 21:11:24.422015 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:24.421989 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" event={"ID":"e76ed6c9-e074-4145-9235-147add5f25b1","Type":"ContainerStarted","Data":"aa32b4ad15a7481e3b9e45901e21412e4cb48dabfe68ba27e3592dac78379804"} Nov 03 21:11:24.442153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:24.442104 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" podStartSLOduration=1.442090264 podStartE2EDuration="1.442090264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:24.440192864 +0000 UTC m=+4193.970538536" watchObservedRunningTime="2026-11-03 21:11:24.442090264 +0000 UTC m=+4193.972435935" Nov 03 21:11:25.429940 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:25.429900 5815 generic.go:356] "Generic (PLEG): container finished" podID="e76ed6c9-e074-4145-9235-147add5f25b1" containerID="a60630a3c422920d71db8f3efaad72d47ae13c11da3c48cd8a5adfcd9bfaa26e" exitCode=0 Nov 03 21:11:25.430402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:25.429997 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" event={"ID":"e76ed6c9-e074-4145-9235-147add5f25b1","Type":"ContainerDied","Data":"a60630a3c422920d71db8f3efaad72d47ae13c11da3c48cd8a5adfcd9bfaa26e"} Nov 03 21:11:26.575381 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.575352 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:26.673624 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.673593 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw\" (UniqueName: \"kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw\") pod \"e76ed6c9-e074-4145-9235-147add5f25b1\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " Nov 03 21:11:26.673828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.673702 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca\") pod \"e76ed6c9-e074-4145-9235-147add5f25b1\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " Nov 03 21:11:26.673828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.673793 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca\") pod \"e76ed6c9-e074-4145-9235-147add5f25b1\" (UID: \"e76ed6c9-e074-4145-9235-147add5f25b1\") " Nov 03 21:11:26.674164 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.674099 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca" pod "e76ed6c9-e074-4145-9235-147add5f25b1" (UID: "e76ed6c9-e074-4145-9235-147add5f25b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:26.675983 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.675956 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca" pod "e76ed6c9-e074-4145-9235-147add5f25b1" (UID: "e76ed6c9-e074-4145-9235-147add5f25b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:26.676099 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.675979 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw" pod "e76ed6c9-e074-4145-9235-147add5f25b1" (UID: "e76ed6c9-e074-4145-9235-147add5f25b1"). InnerVolumeSpecName "kube-api-access-xx6fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:26.775168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.775135 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:26.775168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.775165 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e76ed6c9-e074-4145-9235-147add5f25b1-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:26.775168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:26.775174 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx6fw\" (UniqueName: \"kubernetes.io/projected/e76ed6c9-e074-4145-9235-147add5f25b1-kube-api-access-xx6fw\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:27.450691 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:27.450654 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" event={"ID":"e76ed6c9-e074-4145-9235-147add5f25b1","Type":"ContainerDied","Data":"aa32b4ad15a7481e3b9e45901e21412e4cb48dabfe68ba27e3592dac78379804"} Nov 03 21:11:27.450889 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:27.450700 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6" Nov 03 21:11:27.450889 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:27.450737 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa32b4ad15a7481e3b9e45901e21412e4cb48dabfe68ba27e3592dac78379804" Nov 03 21:11:34.276037 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:34.275989 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:11:36.320158 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:36.320127 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:11:36.326317 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:36.326294 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:11:36.337411 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:36.337393 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:11:36.343683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:36.343663 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:11:42.280554 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:42.280522 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:11:43.852287 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:11:43.852249 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:11:47.790539 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:47.790490 5815 scope.go:122] "RemoveContainer" containerID="22c73fa351ce06cb15c242b69b86c0c9c81bcb5cd3b1da235c4e6beeabfd070b" Nov 03 21:11:47.800057 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:11:47.800039 5815 scope.go:122] "RemoveContainer" containerID="6d9ad55eb00e153e74be32789906628818bdcff13f8e7154d0e2c4e2b58b8c60" Nov 03 21:12:13.852079 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:12:13.852003 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:12:14.865489 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:12:14.865449 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:12:18.375721 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:12:18.375683 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:12:32.585828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:12:32.585793 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:12:38.566715 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:12:38.566673 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:12:43.851073 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:12:43.851030 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:12:54.451688 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:12:54.451648 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:13:10.137754 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:10.137714 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:13:13.951236 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:13:13.951156 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:13:21.864318 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:21.864282 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:13:29.556084 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:29.556049 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:13:37.328611 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:37.328572 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:13:39.481593 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:39.480622 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:13:39.481593 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:39.480707 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:13:42.417551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:42.417351 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:13:43.852227 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:13:43.852186 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:13:55.447779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:13:55.447746 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:14:13.850306 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:14:13.850269 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:14:42.366923 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:14:42.366886 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:14:43.852908 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:14:43.852868 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:15:13.849498 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:15:13.849409 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:15:43.852893 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:15:43.852856 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:16:13.850563 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:16:13.850493 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:16:18.174967 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:18.174929 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:16:24.274041 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:24.274006 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:16:27.039531 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:27.038851 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6"] Nov 03 21:16:27.044534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:27.042217 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c7f157a-6zlp6"] Nov 03 21:16:28.584605 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:28.584495 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76ed6c9-e074-4145-9235-147add5f25b1" path="/var/lib/kubelet/pods/e76ed6c9-e074-4145-9235-147add5f25b1/volumes" Nov 03 21:16:28.750620 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:28.750582 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:16:36.245107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:36.245072 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:16:36.371685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:36.371654 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:16:36.377743 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:36.377722 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:16:36.382303 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:36.382286 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:16:36.388391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:36.388371 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:16:40.672238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:40.672023 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:16:43.853265 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:16:43.853231 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:16:53.476544 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:16:53.476498 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:17:06.269610 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:06.269579 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:17:09.482658 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:09.482604 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:17:09.483314 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:09.483265 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:17:13.850815 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:17:13.850778 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:17:30.486262 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:30.486225 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:17:40.768181 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:40.768148 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:17:43.858958 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:17:43.858903 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:17:46.186450 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:46.186413 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:17:47.868033 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:17:47.867995 5815 scope.go:122] "RemoveContainer" containerID="a60630a3c422920d71db8f3efaad72d47ae13c11da3c48cd8a5adfcd9bfaa26e" Nov 03 21:18:01.571660 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:01.571626 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:05.906541 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:05.905191 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:09.633074 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:09.632984 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:13.850047 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:18:13.850010 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:18:20.476780 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:20.476741 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:34.175648 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:34.175614 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:38.462373 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:38.462337 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:39.481663 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:39.481603 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-gqpt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:18:39.482478 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:39.482442 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-gqpt8" podUID="ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:18:41.984172 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:41.984134 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:18:43.852742 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:18:43.852694 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:18:59.581010 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:18:59.580969 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:19:04.637534 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:19:04.632344 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:19:13.851138 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:19:13.851097 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:19:18.240446 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:19:18.240411 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:19:22.565705 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:19:22.565669 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:19:28.036816 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:19:28.036779 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:19:43.851439 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:19:43.851349 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:19:46.346562 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:19:46.346523 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:19:56.641432 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:19:56.641397 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:20:12.963146 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:20:12.963106 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:20:13.919028 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:20:13.918884 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:20:18.134772 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:20:18.134733 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:20:24.159653 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:20:24.159620 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:20:40.380246 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:20:40.380211 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:20:43.853141 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:20:43.853104 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:20:56.265649 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:20:56.265614 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:21:13.849966 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:21:13.849888 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:21:21.360628 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:21:21.360597 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:21:29.437617 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:21:29.437577 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:21:36.420108 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:21:36.420080 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:21:36.431364 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:21:36.431337 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:21:36.431543 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:21:36.431422 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:21:36.436993 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:21:36.436975 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:21:43.850861 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:21:43.850822 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:22:02.153532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:02.151940 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:22:07.587086 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:07.582438 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:22:13.850337 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:22:13.850301 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:22:20.560106 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:20.560052 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:22:35.770835 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:35.770768 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh"] Nov 03 21:22:35.771463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:35.770986 5815 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" podUID="e79d3bb8-3533-4def-a7d4-2fba2b589a0f" containerName="manager" containerID="cri-o://831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8" gracePeriod=10 Nov 03 21:22:36.038799 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.038772 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 21:22:36.213448 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.213409 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume\") pod \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " Nov 03 21:22:36.213712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.213690 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl\" (UniqueName: \"kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl\") pod \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\" (UID: \"e79d3bb8-3533-4def-a7d4-2fba2b589a0f\") " Nov 03 21:22:36.213927 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.213904 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume" pod "e79d3bb8-3533-4def-a7d4-2fba2b589a0f" (UID: "e79d3bb8-3533-4def-a7d4-2fba2b589a0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:22:36.214046 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.214026 5815 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-extensions-socket-volume\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:36.215877 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.215856 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl" pod "e79d3bb8-3533-4def-a7d4-2fba2b589a0f" (UID: "e79d3bb8-3533-4def-a7d4-2fba2b589a0f"). InnerVolumeSpecName "kube-api-access-kmdgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:36.315231 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.315169 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmdgl\" (UniqueName: \"kubernetes.io/projected/e79d3bb8-3533-4def-a7d4-2fba2b589a0f-kube-api-access-kmdgl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:36.818795 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.818756 5815 generic.go:356] "Generic (PLEG): container finished" podID="e79d3bb8-3533-4def-a7d4-2fba2b589a0f" containerID="831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8" exitCode=0 Nov 03 21:22:36.819238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.818799 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" event={"ID":"e79d3bb8-3533-4def-a7d4-2fba2b589a0f","Type":"ContainerDied","Data":"831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8"} Nov 03 21:22:36.819238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.818818 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" Nov 03 21:22:36.819238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.818875 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh" event={"ID":"e79d3bb8-3533-4def-a7d4-2fba2b589a0f","Type":"ContainerDied","Data":"91ec1d1fef25cafe132f1d2bd47c7120b2d4a0130ccbb43dfc6565abfcfe79fd"} Nov 03 21:22:36.819238 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.818897 5815 scope.go:122] "RemoveContainer" containerID="831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8" Nov 03 21:22:36.833168 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.833153 5815 scope.go:122] "RemoveContainer" containerID="831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8" Nov 03 21:22:36.833416 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:22:36.833394 5815 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8\": container with ID starting with 831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8 not found: ID does not exist" containerID="831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8" Nov 03 21:22:36.833513 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.833430 5815 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8"} err="failed to get container status \"831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8\": rpc error: code = NotFound desc = could not find container \"831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8\": container with ID starting with 831621d72db585c32df990d85da07246d2eb4baa0ae5ca7a31ef3de9426f9ab8 not found: ID does not exist" Nov 03 21:22:36.839191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.839169 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh"] Nov 03 21:22:36.843250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:36.843201 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-k4psh"] Nov 03 21:22:38.584301 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:22:38.584224 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79d3bb8-3533-4def-a7d4-2fba2b589a0f" path="/var/lib/kubelet/pods/e79d3bb8-3533-4def-a7d4-2fba2b589a0f/volumes" Nov 03 21:22:43.850444 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:22:43.850410 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:23:13.850317 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:23:13.850280 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:23:43.849822 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:23:43.849785 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:24:03.565739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:03.565706 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:24:13.850418 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:24:13.850337 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:24:18.716333 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:18.716301 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:24:24.363498 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:24.363449 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:24:27.802626 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:27.802591 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:24:43.851604 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:24:43.851566 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:24:45.941247 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:45.941213 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:24:55.668165 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668128 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2"] Nov 03 21:24:55.668627 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668524 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e79d3bb8-3533-4def-a7d4-2fba2b589a0f" containerName="manager" Nov 03 21:24:55.668627 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668536 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79d3bb8-3533-4def-a7d4-2fba2b589a0f" containerName="manager" Nov 03 21:24:55.668627 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668552 5815 cpu_manager.go:414] "containerMap: removing container" podUID="e76ed6c9-e074-4145-9235-147add5f25b1" containerName="revoke-keys" Nov 03 21:24:55.668627 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668557 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76ed6c9-e074-4145-9235-147add5f25b1" containerName="revoke-keys" Nov 03 21:24:55.668801 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668637 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79d3bb8-3533-4def-a7d4-2fba2b589a0f" containerName="manager" Nov 03 21:24:55.668801 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.668647 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76ed6c9-e074-4145-9235-147add5f25b1" containerName="revoke-keys" Nov 03 21:24:55.671865 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.671846 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.678183 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.678156 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:55.679869 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.679844 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:24:55.682049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.682020 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2"] Nov 03 21:24:55.764161 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.764133 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.764306 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.764170 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwp2\" (UniqueName: \"kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.764411 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.764371 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.865473 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.865440 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.865651 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.865500 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.865651 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.865558 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwp2\" (UniqueName: \"kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.866189 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.866168 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.868036 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.868009 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.878302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.878281 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwp2\" (UniqueName: \"kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2\") pod \"maas-api-revoke-sub-3a6e4be7-844p2\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:55.986254 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:55.986232 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:56.265551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:56.265518 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2"] Nov 03 21:24:56.267986 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:24:56.267955 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1c0986_8eb3_4d0b_9b31_9b785c7fab4a.slice/crio-e6d75b067bc34a4b80d378d58a8ccdf183cd826f62a4d56983dce4e1c3bca0f9 WatchSource:0}: Error finding container e6d75b067bc34a4b80d378d58a8ccdf183cd826f62a4d56983dce4e1c3bca0f9: Status 404 returned error can't find the container with id e6d75b067bc34a4b80d378d58a8ccdf183cd826f62a4d56983dce4e1c3bca0f9 Nov 03 21:24:56.674087 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:56.674019 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" event={"ID":"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a","Type":"ContainerStarted","Data":"a587f3155edb915508bbfc963e85b105f95c790524d01ab5ebef5a8754dbf49f"} Nov 03 21:24:56.674087 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:56.674066 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" event={"ID":"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a","Type":"ContainerStarted","Data":"e6d75b067bc34a4b80d378d58a8ccdf183cd826f62a4d56983dce4e1c3bca0f9"} Nov 03 21:24:56.694262 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:56.694220 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" podStartSLOduration=1.69420554 podStartE2EDuration="1.69420554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:56.692616161 +0000 UTC m=+5006.222961833" watchObservedRunningTime="2026-11-03 21:24:56.69420554 +0000 UTC m=+5006.224551211" Nov 03 21:24:57.680764 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:57.680724 5815 generic.go:356] "Generic (PLEG): container finished" podID="3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" containerID="a587f3155edb915508bbfc963e85b105f95c790524d01ab5ebef5a8754dbf49f" exitCode=0 Nov 03 21:24:57.681165 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:57.680810 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" event={"ID":"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a","Type":"ContainerDied","Data":"a587f3155edb915508bbfc963e85b105f95c790524d01ab5ebef5a8754dbf49f"} Nov 03 21:24:58.835706 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.835683 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:58.990323 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.990300 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca\") pod \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " Nov 03 21:24:58.990471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.990380 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2\" (UniqueName: \"kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2\") pod \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " Nov 03 21:24:58.990471 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.990398 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca\") pod \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\" (UID: \"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a\") " Nov 03 21:24:58.990677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.990654 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca" pod "3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" (UID: "3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:58.992884 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.992860 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca" pod "3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" (UID: "3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:58.992991 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:58.992887 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2" pod "3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" (UID: "3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a"). InnerVolumeSpecName "kube-api-access-dgwp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:59.091352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:59.091325 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgwp2\" (UniqueName: \"kubernetes.io/projected/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-kube-api-access-dgwp2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:59.091352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:59.091350 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:59.091498 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:59.091360 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:59.693683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:59.693657 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" Nov 03 21:24:59.693862 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:59.693657 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2" event={"ID":"3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a","Type":"ContainerDied","Data":"e6d75b067bc34a4b80d378d58a8ccdf183cd826f62a4d56983dce4e1c3bca0f9"} Nov 03 21:24:59.693862 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:24:59.693786 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d75b067bc34a4b80d378d58a8ccdf183cd826f62a4d56983dce4e1c3bca0f9" Nov 03 21:25:04.779017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.778989 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9"] Nov 03 21:25:04.779426 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.779395 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" containerName="revoke-keys" Nov 03 21:25:04.779426 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.779408 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" containerName="revoke-keys" Nov 03 21:25:04.779579 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.779479 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" containerName="revoke-keys" Nov 03 21:25:04.782797 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.782771 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:04.790652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.789053 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:25:04.791137 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.791033 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:25:04.794053 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.794029 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9"] Nov 03 21:25:04.940370 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.940341 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:04.940616 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.940478 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7lm\" (UniqueName: \"kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:04.940616 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:04.940594 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.041940 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.041864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7lm\" (UniqueName: \"kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.041940 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.041927 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.042188 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.041951 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.042746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.042725 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.044487 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.044467 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.054177 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.054155 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7lm\" (UniqueName: \"kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm\") pod \"maas-api-revoke-sub-3400d899-278b9\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.097473 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.097448 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:05.372886 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.372859 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9"] Nov 03 21:25:05.374880 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:25:05.374852 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42972838_3a85_4746_833e_f73b9d3f478e.slice/crio-1d2930b35ca299394375a884da0f4d88c14c38933f57178b8f629f7a124b70a3 WatchSource:0}: Error finding container 1d2930b35ca299394375a884da0f4d88c14c38933f57178b8f629f7a124b70a3: Status 404 returned error can't find the container with id 1d2930b35ca299394375a884da0f4d88c14c38933f57178b8f629f7a124b70a3 Nov 03 21:25:05.734058 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.734022 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" event={"ID":"42972838-3a85-4746-833e-f73b9d3f478e","Type":"ContainerStarted","Data":"0b877a8d59e6fe4003bc88d58b592755a283e5b0549c54e76d9b2879a47a181a"} Nov 03 21:25:05.734208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.734083 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" event={"ID":"42972838-3a85-4746-833e-f73b9d3f478e","Type":"ContainerStarted","Data":"1d2930b35ca299394375a884da0f4d88c14c38933f57178b8f629f7a124b70a3"} Nov 03 21:25:05.753092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:05.753049 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" podStartSLOduration=1.753036233 podStartE2EDuration="1.753036233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:05.752831384 +0000 UTC m=+5015.283177069" watchObservedRunningTime="2026-11-03 21:25:05.753036233 +0000 UTC m=+5015.283381883" Nov 03 21:25:06.741206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:06.741169 5815 generic.go:356] "Generic (PLEG): container finished" podID="42972838-3a85-4746-833e-f73b9d3f478e" containerID="0b877a8d59e6fe4003bc88d58b592755a283e5b0549c54e76d9b2879a47a181a" exitCode=0 Nov 03 21:25:06.741694 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:06.741230 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" event={"ID":"42972838-3a85-4746-833e-f73b9d3f478e","Type":"ContainerDied","Data":"0b877a8d59e6fe4003bc88d58b592755a283e5b0549c54e76d9b2879a47a181a"} Nov 03 21:25:07.880163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.880140 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:07.967416 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.967388 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca\") pod \"42972838-3a85-4746-833e-f73b9d3f478e\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " Nov 03 21:25:07.967614 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.967454 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca\") pod \"42972838-3a85-4746-833e-f73b9d3f478e\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " Nov 03 21:25:07.967614 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.967569 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm\" (UniqueName: \"kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm\") pod \"42972838-3a85-4746-833e-f73b9d3f478e\" (UID: \"42972838-3a85-4746-833e-f73b9d3f478e\") " Nov 03 21:25:07.967832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.967809 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca" pod "42972838-3a85-4746-833e-f73b9d3f478e" (UID: "42972838-3a85-4746-833e-f73b9d3f478e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:07.967957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.967874 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:07.969687 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.969666 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca" pod "42972838-3a85-4746-833e-f73b9d3f478e" (UID: "42972838-3a85-4746-833e-f73b9d3f478e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:07.969775 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:07.969756 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm" pod "42972838-3a85-4746-833e-f73b9d3f478e" (UID: "42972838-3a85-4746-833e-f73b9d3f478e"). InnerVolumeSpecName "kube-api-access-5q7lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:08.069188 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:08.069134 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7lm\" (UniqueName: \"kubernetes.io/projected/42972838-3a85-4746-833e-f73b9d3f478e-kube-api-access-5q7lm\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:08.069188 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:08.069157 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42972838-3a85-4746-833e-f73b9d3f478e-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:08.752728 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:08.752691 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" event={"ID":"42972838-3a85-4746-833e-f73b9d3f478e","Type":"ContainerDied","Data":"1d2930b35ca299394375a884da0f4d88c14c38933f57178b8f629f7a124b70a3"} Nov 03 21:25:08.752728 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:08.752716 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9" Nov 03 21:25:08.753010 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:08.752752 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d2930b35ca299394375a884da0f4d88c14c38933f57178b8f629f7a124b70a3" Nov 03 21:25:13.850729 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:25:13.850689 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:25:15.736500 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:15.736466 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:25:22.202631 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:22.202584 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:25:27.767293 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.767253 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8"] Nov 03 21:25:27.767874 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.767854 5815 cpu_manager.go:414] "containerMap: removing container" podUID="42972838-3a85-4746-833e-f73b9d3f478e" containerName="revoke-keys" Nov 03 21:25:27.767927 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.767877 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42972838-3a85-4746-833e-f73b9d3f478e" containerName="revoke-keys" Nov 03 21:25:27.768041 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.768029 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="42972838-3a85-4746-833e-f73b9d3f478e" containerName="revoke-keys" Nov 03 21:25:27.771753 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.771731 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:27.776566 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.776471 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:25:27.776566 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.776540 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:25:27.777951 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.777928 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8"] Nov 03 21:25:27.901690 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.901658 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:27.901885 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.901797 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:27.901957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:27.901904 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg7ts\" (UniqueName: \"kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.002615 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.002591 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.002779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.002650 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.002779 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.002681 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg7ts\" (UniqueName: \"kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.003413 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.003394 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.005293 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.005273 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.015106 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.015085 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg7ts\" (UniqueName: \"kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts\") pod \"maas-api-revoke-sub-91e55dcb-764m8\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.087139 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.087088 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:28.362700 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.362674 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8"] Nov 03 21:25:28.364730 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:25:28.364704 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df23b8b_3a84_48bc_aad8_fe9961216d50.slice/crio-758bad41cb3f6fd2086e3159749bbbb43df6f2ca09949e2996d9a09740487868 WatchSource:0}: Error finding container 758bad41cb3f6fd2086e3159749bbbb43df6f2ca09949e2996d9a09740487868: Status 404 returned error can't find the container with id 758bad41cb3f6fd2086e3159749bbbb43df6f2ca09949e2996d9a09740487868 Nov 03 21:25:28.885782 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.885747 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" event={"ID":"4df23b8b-3a84-48bc-aad8-fe9961216d50","Type":"ContainerStarted","Data":"9ec0623d8380b3042bc731501850ffd5d97b6b8ab9381cf1050997321a9d3a37"} Nov 03 21:25:28.886202 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.885799 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" event={"ID":"4df23b8b-3a84-48bc-aad8-fe9961216d50","Type":"ContainerStarted","Data":"758bad41cb3f6fd2086e3159749bbbb43df6f2ca09949e2996d9a09740487868"} Nov 03 21:25:28.905988 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:28.905943 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" podStartSLOduration=1.9059300700000001 podStartE2EDuration="1.90593007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:28.903580739 +0000 UTC m=+5038.433926412" watchObservedRunningTime="2026-11-03 21:25:28.90593007 +0000 UTC m=+5038.436275742" Nov 03 21:25:29.892791 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:29.892753 5815 generic.go:356] "Generic (PLEG): container finished" podID="4df23b8b-3a84-48bc-aad8-fe9961216d50" containerID="9ec0623d8380b3042bc731501850ffd5d97b6b8ab9381cf1050997321a9d3a37" exitCode=0 Nov 03 21:25:29.893188 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:29.892836 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" event={"ID":"4df23b8b-3a84-48bc-aad8-fe9961216d50","Type":"ContainerDied","Data":"9ec0623d8380b3042bc731501850ffd5d97b6b8ab9381cf1050997321a9d3a37"} Nov 03 21:25:31.036824 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.036798 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:31.131009 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.130982 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts\" (UniqueName: \"kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts\") pod \"4df23b8b-3a84-48bc-aad8-fe9961216d50\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " Nov 03 21:25:31.131144 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.131076 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca\") pod \"4df23b8b-3a84-48bc-aad8-fe9961216d50\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " Nov 03 21:25:31.131144 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.131110 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca\") pod \"4df23b8b-3a84-48bc-aad8-fe9961216d50\" (UID: \"4df23b8b-3a84-48bc-aad8-fe9961216d50\") " Nov 03 21:25:31.131439 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.131412 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca" pod "4df23b8b-3a84-48bc-aad8-fe9961216d50" (UID: "4df23b8b-3a84-48bc-aad8-fe9961216d50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:31.133388 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.133363 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts" pod "4df23b8b-3a84-48bc-aad8-fe9961216d50" (UID: "4df23b8b-3a84-48bc-aad8-fe9961216d50"). InnerVolumeSpecName "kube-api-access-xg7ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:31.133488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.133410 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca" pod "4df23b8b-3a84-48bc-aad8-fe9961216d50" (UID: "4df23b8b-3a84-48bc-aad8-fe9961216d50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:31.232556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.232526 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:31.232556 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.232554 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df23b8b-3a84-48bc-aad8-fe9961216d50-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:31.232733 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.232572 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg7ts\" (UniqueName: \"kubernetes.io/projected/4df23b8b-3a84-48bc-aad8-fe9961216d50-kube-api-access-xg7ts\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:31.907103 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.907064 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" event={"ID":"4df23b8b-3a84-48bc-aad8-fe9961216d50","Type":"ContainerDied","Data":"758bad41cb3f6fd2086e3159749bbbb43df6f2ca09949e2996d9a09740487868"} Nov 03 21:25:31.907293 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.907115 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758bad41cb3f6fd2086e3159749bbbb43df6f2ca09949e2996d9a09740487868" Nov 03 21:25:31.907293 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:31.907073 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8" Nov 03 21:25:38.654309 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:38.654232 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:25:43.862035 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:25:43.861997 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:25:45.108468 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:45.108434 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:25:55.183340 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.183307 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7"] Nov 03 21:25:55.183860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.183752 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4df23b8b-3a84-48bc-aad8-fe9961216d50" containerName="revoke-keys" Nov 03 21:25:55.183860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.183765 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df23b8b-3a84-48bc-aad8-fe9961216d50" containerName="revoke-keys" Nov 03 21:25:55.183941 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.183861 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df23b8b-3a84-48bc-aad8-fe9961216d50" containerName="revoke-keys" Nov 03 21:25:55.187082 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.187065 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.194540 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.194461 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:25:55.194652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.194542 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:25:55.196170 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.196141 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7"] Nov 03 21:25:55.316092 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.316065 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.316260 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.316099 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.316260 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.316131 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q9r\" (UniqueName: \"kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.419643 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.419608 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.419828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.419646 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.419828 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.419676 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q9r\" (UniqueName: \"kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.420554 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.420533 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.422336 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.422314 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.432912 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.432885 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q9r\" (UniqueName: \"kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r\") pod \"maas-api-revoke-sub-2deb6417-r9hs7\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.502021 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.501997 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:55.782163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:55.782115 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7"] Nov 03 21:25:55.783800 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:25:55.783773 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22a9d03_82a9_4d24_b85c_4b600817845a.slice/crio-9bb4a5523b28529e8d043b4dcf1640d66d3dd790e5a8ced92de6588596e8623c WatchSource:0}: Error finding container 9bb4a5523b28529e8d043b4dcf1640d66d3dd790e5a8ced92de6588596e8623c: Status 404 returned error can't find the container with id 9bb4a5523b28529e8d043b4dcf1640d66d3dd790e5a8ced92de6588596e8623c Nov 03 21:25:56.085297 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:56.085233 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" event={"ID":"c22a9d03-82a9-4d24-b85c-4b600817845a","Type":"ContainerStarted","Data":"016b1172d58b4dffec4febdf6edc701078def20db22accf049f658189a3028a8"} Nov 03 21:25:56.085297 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:56.085287 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" event={"ID":"c22a9d03-82a9-4d24-b85c-4b600817845a","Type":"ContainerStarted","Data":"9bb4a5523b28529e8d043b4dcf1640d66d3dd790e5a8ced92de6588596e8623c"} Nov 03 21:25:56.105795 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:56.105752 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" podStartSLOduration=1.105737343 podStartE2EDuration="1.105737343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:56.10393185 +0000 UTC m=+5065.634277522" watchObservedRunningTime="2026-11-03 21:25:56.105737343 +0000 UTC m=+5065.636083015" Nov 03 21:25:57.093295 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:57.093254 5815 generic.go:356] "Generic (PLEG): container finished" podID="c22a9d03-82a9-4d24-b85c-4b600817845a" containerID="016b1172d58b4dffec4febdf6edc701078def20db22accf049f658189a3028a8" exitCode=0 Nov 03 21:25:57.093787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:57.093362 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" event={"ID":"c22a9d03-82a9-4d24-b85c-4b600817845a","Type":"ContainerDied","Data":"016b1172d58b4dffec4febdf6edc701078def20db22accf049f658189a3028a8"} Nov 03 21:25:58.244340 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.244316 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:58.344896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.344870 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r\" (UniqueName: \"kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r\") pod \"c22a9d03-82a9-4d24-b85c-4b600817845a\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " Nov 03 21:25:58.345049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.344990 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca\") pod \"c22a9d03-82a9-4d24-b85c-4b600817845a\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " Nov 03 21:25:58.345095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.345052 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca\") pod \"c22a9d03-82a9-4d24-b85c-4b600817845a\" (UID: \"c22a9d03-82a9-4d24-b85c-4b600817845a\") " Nov 03 21:25:58.345402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.345378 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca" pod "c22a9d03-82a9-4d24-b85c-4b600817845a" (UID: "c22a9d03-82a9-4d24-b85c-4b600817845a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:58.347279 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.347260 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r" pod "c22a9d03-82a9-4d24-b85c-4b600817845a" (UID: "c22a9d03-82a9-4d24-b85c-4b600817845a"). InnerVolumeSpecName "kube-api-access-74q9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:58.347279 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.347272 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca" pod "c22a9d03-82a9-4d24-b85c-4b600817845a" (UID: "c22a9d03-82a9-4d24-b85c-4b600817845a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:58.446744 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.446717 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:58.446744 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.446745 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22a9d03-82a9-4d24-b85c-4b600817845a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:58.446939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:58.446754 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74q9r\" (UniqueName: \"kubernetes.io/projected/c22a9d03-82a9-4d24-b85c-4b600817845a-kube-api-access-74q9r\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:59.106169 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:59.106138 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" Nov 03 21:25:59.106169 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:59.106155 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2deb6417-r9hs7" event={"ID":"c22a9d03-82a9-4d24-b85c-4b600817845a","Type":"ContainerDied","Data":"9bb4a5523b28529e8d043b4dcf1640d66d3dd790e5a8ced92de6588596e8623c"} Nov 03 21:25:59.106401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:25:59.106198 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb4a5523b28529e8d043b4dcf1640d66d3dd790e5a8ced92de6588596e8623c" Nov 03 21:26:05.397670 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.397634 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl"] Nov 03 21:26:05.398120 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.398060 5815 cpu_manager.go:414] "containerMap: removing container" podUID="c22a9d03-82a9-4d24-b85c-4b600817845a" containerName="revoke-keys" Nov 03 21:26:05.398120 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.398072 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22a9d03-82a9-4d24-b85c-4b600817845a" containerName="revoke-keys" Nov 03 21:26:05.398231 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.398144 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22a9d03-82a9-4d24-b85c-4b600817845a" containerName="revoke-keys" Nov 03 21:26:05.401687 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.401660 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.408896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.408737 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:26:05.408896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.408739 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:26:05.412299 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.410538 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl"] Nov 03 21:26:05.503103 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.503069 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.503299 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.503115 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.503383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.503288 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drz8\" (UniqueName: \"kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.603852 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.603825 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.603980 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.603867 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.603980 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.603961 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drz8\" (UniqueName: \"kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.604548 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.604499 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.606422 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.606398 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.616302 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.616278 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drz8\" (UniqueName: \"kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8\") pod \"maas-api-revoke-sub-daee2067-xjkfl\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.717325 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.717302 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:05.995006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:05.994979 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl"] Nov 03 21:26:05.997397 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:26:05.997373 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49444382_aa13_4ee0_8009_28e4f5892c5a.slice/crio-c3bccac0c0bbf94a94143fb19c1beb867a5e57ddea560252e96018a15ca9656e WatchSource:0}: Error finding container c3bccac0c0bbf94a94143fb19c1beb867a5e57ddea560252e96018a15ca9656e: Status 404 returned error can't find the container with id c3bccac0c0bbf94a94143fb19c1beb867a5e57ddea560252e96018a15ca9656e Nov 03 21:26:06.143690 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:06.143643 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:26:06.146559 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:06.146537 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" event={"ID":"49444382-aa13-4ee0-8009-28e4f5892c5a","Type":"ContainerStarted","Data":"c3bccac0c0bbf94a94143fb19c1beb867a5e57ddea560252e96018a15ca9656e"} Nov 03 21:26:07.154340 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:07.154264 5815 generic.go:356] "Generic (PLEG): container finished" podID="49444382-aa13-4ee0-8009-28e4f5892c5a" containerID="e276cc5c131aeb926c7d85af7b25838e7693ad7cb6c6ab4dacb751bda31c10d4" exitCode=0 Nov 03 21:26:07.154747 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:07.154340 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" event={"ID":"49444382-aa13-4ee0-8009-28e4f5892c5a","Type":"ContainerDied","Data":"e276cc5c131aeb926c7d85af7b25838e7693ad7cb6c6ab4dacb751bda31c10d4"} Nov 03 21:26:08.301836 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.301810 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:08.431089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.431025 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8\" (UniqueName: \"kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8\") pod \"49444382-aa13-4ee0-8009-28e4f5892c5a\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " Nov 03 21:26:08.431089 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.431080 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca\") pod \"49444382-aa13-4ee0-8009-28e4f5892c5a\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " Nov 03 21:26:08.431313 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.431163 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca\") pod \"49444382-aa13-4ee0-8009-28e4f5892c5a\" (UID: \"49444382-aa13-4ee0-8009-28e4f5892c5a\") " Nov 03 21:26:08.431604 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.431580 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca" pod "49444382-aa13-4ee0-8009-28e4f5892c5a" (UID: "49444382-aa13-4ee0-8009-28e4f5892c5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:08.433326 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.433299 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca" pod "49444382-aa13-4ee0-8009-28e4f5892c5a" (UID: "49444382-aa13-4ee0-8009-28e4f5892c5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:08.433550 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.433499 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8" pod "49444382-aa13-4ee0-8009-28e4f5892c5a" (UID: "49444382-aa13-4ee0-8009-28e4f5892c5a"). InnerVolumeSpecName "kube-api-access-4drz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:08.532415 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.532390 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drz8\" (UniqueName: \"kubernetes.io/projected/49444382-aa13-4ee0-8009-28e4f5892c5a-kube-api-access-4drz8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:08.532415 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.532414 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:08.532584 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:08.532424 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49444382-aa13-4ee0-8009-28e4f5892c5a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:09.166318 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:09.166283 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" Nov 03 21:26:09.166547 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:09.166315 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee2067-xjkfl" event={"ID":"49444382-aa13-4ee0-8009-28e4f5892c5a","Type":"ContainerDied","Data":"c3bccac0c0bbf94a94143fb19c1beb867a5e57ddea560252e96018a15ca9656e"} Nov 03 21:26:09.166547 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:09.166368 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3bccac0c0bbf94a94143fb19c1beb867a5e57ddea560252e96018a15ca9656e" Nov 03 21:26:13.850689 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:26:13.850653 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:26:16.339043 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:16.339006 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:26:22.702492 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:22.702456 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:26:28.120879 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:28.120846 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:26:35.678783 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.678749 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q"] Nov 03 21:26:35.679340 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.679320 5815 cpu_manager.go:414] "containerMap: removing container" podUID="49444382-aa13-4ee0-8009-28e4f5892c5a" containerName="revoke-keys" Nov 03 21:26:35.679433 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.679343 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49444382-aa13-4ee0-8009-28e4f5892c5a" containerName="revoke-keys" Nov 03 21:26:35.679500 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.679469 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="49444382-aa13-4ee0-8009-28e4f5892c5a" containerName="revoke-keys" Nov 03 21:26:35.682852 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.682832 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.690488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.690469 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:26:35.690621 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.690576 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:26:35.691342 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.691320 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q"] Nov 03 21:26:35.754764 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.754732 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.754921 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.754801 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4gr\" (UniqueName: \"kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.754921 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.754878 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.855898 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.855864 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.856076 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.855951 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.856076 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.856046 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4gr\" (UniqueName: \"kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.856545 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.856499 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.858587 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.858568 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.868683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.868664 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4gr\" (UniqueName: \"kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr\") pod \"maas-api-revoke-sub-f829c667-qzg9q\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:35.999164 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:35.999134 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:36.276250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:36.276226 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q"] Nov 03 21:26:36.278483 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:26:36.278455 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccac898a_c58d_4879_8b4f_c78b67c2ff6a.slice/crio-896468b92f9cb92c7406c912e393ea725a31d6c8aae3595a6aae0bde32a78789 WatchSource:0}: Error finding container 896468b92f9cb92c7406c912e393ea725a31d6c8aae3595a6aae0bde32a78789: Status 404 returned error can't find the container with id 896468b92f9cb92c7406c912e393ea725a31d6c8aae3595a6aae0bde32a78789 Nov 03 21:26:36.365966 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:36.365939 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" event={"ID":"ccac898a-c58d-4879-8b4f-c78b67c2ff6a","Type":"ContainerStarted","Data":"896468b92f9cb92c7406c912e393ea725a31d6c8aae3595a6aae0bde32a78789"} Nov 03 21:26:36.471202 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:36.471182 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:26:36.473630 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:36.473609 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:26:36.476992 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:36.476972 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:26:36.479922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:36.479904 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt8zf_849c8451-1aac-44aa-a685-28b03801682e/ovn-acl-logging/0.log" Nov 03 21:26:37.373197 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:37.373127 5815 generic.go:356] "Generic (PLEG): container finished" podID="ccac898a-c58d-4879-8b4f-c78b67c2ff6a" containerID="24a091624aaad659047cf0c36e16e4a245a49a13e52187f9f29cb4d1d8e0274f" exitCode=0 Nov 03 21:26:37.373606 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:37.373197 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" event={"ID":"ccac898a-c58d-4879-8b4f-c78b67c2ff6a","Type":"ContainerDied","Data":"24a091624aaad659047cf0c36e16e4a245a49a13e52187f9f29cb4d1d8e0274f"} Nov 03 21:26:38.520025 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.520003 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:38.579948 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.579915 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca\") pod \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " Nov 03 21:26:38.580126 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.580001 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr\" (UniqueName: \"kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr\") pod \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " Nov 03 21:26:38.580126 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.580028 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca\") pod \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\" (UID: \"ccac898a-c58d-4879-8b4f-c78b67c2ff6a\") " Nov 03 21:26:38.580494 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.580463 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca" pod "ccac898a-c58d-4879-8b4f-c78b67c2ff6a" (UID: "ccac898a-c58d-4879-8b4f-c78b67c2ff6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:38.582236 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.582207 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca" pod "ccac898a-c58d-4879-8b4f-c78b67c2ff6a" (UID: "ccac898a-c58d-4879-8b4f-c78b67c2ff6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:38.582408 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.582389 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr" pod "ccac898a-c58d-4879-8b4f-c78b67c2ff6a" (UID: "ccac898a-c58d-4879-8b4f-c78b67c2ff6a"). InnerVolumeSpecName "kube-api-access-tq4gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:38.681365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.681313 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:38.681365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.681335 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4gr\" (UniqueName: \"kubernetes.io/projected/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-kube-api-access-tq4gr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:38.681365 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:38.681345 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccac898a-c58d-4879-8b4f-c78b67c2ff6a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:39.386158 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:39.386125 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" Nov 03 21:26:39.386325 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:39.386128 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829c667-qzg9q" event={"ID":"ccac898a-c58d-4879-8b4f-c78b67c2ff6a","Type":"ContainerDied","Data":"896468b92f9cb92c7406c912e393ea725a31d6c8aae3595a6aae0bde32a78789"} Nov 03 21:26:39.386325 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:39.386258 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="896468b92f9cb92c7406c912e393ea725a31d6c8aae3595a6aae0bde32a78789" Nov 03 21:26:43.854716 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:26:43.854678 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:26:45.892757 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.892727 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx"] Nov 03 21:26:45.893163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.893113 5815 cpu_manager.go:414] "containerMap: removing container" podUID="ccac898a-c58d-4879-8b4f-c78b67c2ff6a" containerName="revoke-keys" Nov 03 21:26:45.893163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.893123 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccac898a-c58d-4879-8b4f-c78b67c2ff6a" containerName="revoke-keys" Nov 03 21:26:45.893247 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.893208 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccac898a-c58d-4879-8b4f-c78b67c2ff6a" containerName="revoke-keys" Nov 03 21:26:45.895565 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.895542 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:45.902485 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.902265 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:26:45.902485 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.902401 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:26:45.905611 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.905578 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx"] Nov 03 21:26:45.938770 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.938740 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxql\" (UniqueName: \"kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:45.938947 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.938779 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:45.939009 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:45.938978 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.040453 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.040415 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.040664 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.040556 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.040664 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.040583 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxql\" (UniqueName: \"kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.041136 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.041113 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.043183 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.043157 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.053573 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.053549 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxql\" (UniqueName: \"kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql\") pod \"maas-api-revoke-sub-84d70be0-gmsnx\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.213373 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.213349 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:46.488660 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.488635 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx"] Nov 03 21:26:46.490656 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:26:46.490621 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21977a0_57a9_43b7_97be_67a035b14ebd.slice/crio-0a347594f6fa60d5844de236c2513e1bfa60c3e379346b944f43357f372907dd WatchSource:0}: Error finding container 0a347594f6fa60d5844de236c2513e1bfa60c3e379346b944f43357f372907dd: Status 404 returned error can't find the container with id 0a347594f6fa60d5844de236c2513e1bfa60c3e379346b944f43357f372907dd Nov 03 21:26:46.653452 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:46.653415 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:26:47.437220 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:47.437181 5815 generic.go:356] "Generic (PLEG): container finished" podID="a21977a0-57a9-43b7-97be-67a035b14ebd" containerID="61fb39e390b05a2b46ca731c7fa1c9969739f0ff7100e8c737d6deb936173a7a" exitCode=0 Nov 03 21:26:47.437673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:47.437272 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" event={"ID":"a21977a0-57a9-43b7-97be-67a035b14ebd","Type":"ContainerDied","Data":"61fb39e390b05a2b46ca731c7fa1c9969739f0ff7100e8c737d6deb936173a7a"} Nov 03 21:26:47.437673 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:47.437355 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" event={"ID":"a21977a0-57a9-43b7-97be-67a035b14ebd","Type":"ContainerStarted","Data":"0a347594f6fa60d5844de236c2513e1bfa60c3e379346b944f43357f372907dd"} Nov 03 21:26:48.855133 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:48.855109 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:49.009591 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.008771 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca\") pod \"a21977a0-57a9-43b7-97be-67a035b14ebd\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " Nov 03 21:26:49.009591 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.008860 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql\" (UniqueName: \"kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql\") pod \"a21977a0-57a9-43b7-97be-67a035b14ebd\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " Nov 03 21:26:49.009591 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.008916 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca\") pod \"a21977a0-57a9-43b7-97be-67a035b14ebd\" (UID: \"a21977a0-57a9-43b7-97be-67a035b14ebd\") " Nov 03 21:26:49.014027 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.013998 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca" pod "a21977a0-57a9-43b7-97be-67a035b14ebd" (UID: "a21977a0-57a9-43b7-97be-67a035b14ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:49.037189 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.037135 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql" pod "a21977a0-57a9-43b7-97be-67a035b14ebd" (UID: "a21977a0-57a9-43b7-97be-67a035b14ebd"). InnerVolumeSpecName "kube-api-access-ntxql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:49.043262 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.037285 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca" pod "a21977a0-57a9-43b7-97be-67a035b14ebd" (UID: "a21977a0-57a9-43b7-97be-67a035b14ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:49.121581 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.119089 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:49.121581 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.119127 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntxql\" (UniqueName: \"kubernetes.io/projected/a21977a0-57a9-43b7-97be-67a035b14ebd-kube-api-access-ntxql\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:49.121581 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.119144 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21977a0-57a9-43b7-97be-67a035b14ebd-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:49.455371 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.455271 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" Nov 03 21:26:49.455371 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.455301 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d70be0-gmsnx" event={"ID":"a21977a0-57a9-43b7-97be-67a035b14ebd","Type":"ContainerDied","Data":"0a347594f6fa60d5844de236c2513e1bfa60c3e379346b944f43357f372907dd"} Nov 03 21:26:49.455371 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:49.455347 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a347594f6fa60d5844de236c2513e1bfa60c3e379346b944f43357f372907dd" Nov 03 21:26:56.834856 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:26:56.834814 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:27:01.359313 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:01.359275 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:27:01.836449 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:01.835872 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-h8zwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:27:01.836449 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:01.835998 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-h8zwm" podUID="44953e3d-f118-443e-894d-530fba487260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:27:01.845659 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:01.845635 5815 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xvscb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:27:01.845801 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:01.845684 5815 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xvscb" podUID="da9aeac9-c831-4be9-9aba-39a42918b4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:27:13.856387 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:27:13.856183 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:27:17.401497 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:17.401462 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:27:23.209215 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.209177 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls"] Nov 03 21:27:23.209675 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.209600 5815 cpu_manager.go:414] "containerMap: removing container" podUID="a21977a0-57a9-43b7-97be-67a035b14ebd" containerName="revoke-keys" Nov 03 21:27:23.209675 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.209612 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21977a0-57a9-43b7-97be-67a035b14ebd" containerName="revoke-keys" Nov 03 21:27:23.209755 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.209680 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21977a0-57a9-43b7-97be-67a035b14ebd" containerName="revoke-keys" Nov 03 21:27:23.213079 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.213059 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.219384 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.219356 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:27:23.219830 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.219810 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:27:23.225701 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.225672 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls"] Nov 03 21:27:23.350468 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.350419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.350468 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.350476 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.350750 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.350612 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfhl\" (UniqueName: \"kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.452106 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.452069 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.452293 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.452114 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.452293 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.452186 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfhl\" (UniqueName: \"kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.452741 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.452722 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.454652 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.454629 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.465000 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.464975 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfhl\" (UniqueName: \"kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl\") pod \"maas-api-revoke-sub-227e46b3-nj8ls\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.540654 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.540632 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:23.820949 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:23.820919 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls"] Nov 03 21:27:23.822295 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:27:23.822271 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491c4db8_a01b_4fb4_ba4f_ed5355c17b2c.slice/crio-d947b6b868aa446a04078451e5098fbd3869d8d39212263618424b76747d45fd WatchSource:0}: Error finding container d947b6b868aa446a04078451e5098fbd3869d8d39212263618424b76747d45fd: Status 404 returned error can't find the container with id d947b6b868aa446a04078451e5098fbd3869d8d39212263618424b76747d45fd Nov 03 21:27:24.753290 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:24.753263 5815 generic.go:356] "Generic (PLEG): container finished" podID="491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" containerID="0b79b487b43c73407b3034d933581b54b06b20a4b3b8c45f9eeeddaf7bea137e" exitCode=0 Nov 03 21:27:24.753705 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:24.753359 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" event={"ID":"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c","Type":"ContainerDied","Data":"0b79b487b43c73407b3034d933581b54b06b20a4b3b8c45f9eeeddaf7bea137e"} Nov 03 21:27:24.753705 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:24.753424 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" event={"ID":"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c","Type":"ContainerStarted","Data":"d947b6b868aa446a04078451e5098fbd3869d8d39212263618424b76747d45fd"} Nov 03 21:27:25.896767 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:25.896740 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:26.075852 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.075783 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca\") pod \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " Nov 03 21:27:26.075852 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.075848 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl\" (UniqueName: \"kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl\") pod \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " Nov 03 21:27:26.076105 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.075882 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca\") pod \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\" (UID: \"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c\") " Nov 03 21:27:26.076192 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.076166 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca" pod "491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" (UID: "491c4db8-a01b-4fb4-ba4f-ed5355c17b2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:26.078153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.078130 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca" pod "491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" (UID: "491c4db8-a01b-4fb4-ba4f-ed5355c17b2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:26.078291 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.078270 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl" pod "491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" (UID: "491c4db8-a01b-4fb4-ba4f-ed5355c17b2c"). InnerVolumeSpecName "kube-api-access-5lfhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:26.177361 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.177330 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lfhl\" (UniqueName: \"kubernetes.io/projected/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-kube-api-access-5lfhl\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:26.177361 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.177357 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:26.177569 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.177372 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/491c4db8-a01b-4fb4-ba4f-ed5355c17b2c-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:26.765679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.765646 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" event={"ID":"491c4db8-a01b-4fb4-ba4f-ed5355c17b2c","Type":"ContainerDied","Data":"d947b6b868aa446a04078451e5098fbd3869d8d39212263618424b76747d45fd"} Nov 03 21:27:26.765679 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.765662 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227e46b3-nj8ls" Nov 03 21:27:26.765905 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:26.765691 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d947b6b868aa446a04078451e5098fbd3869d8d39212263618424b76747d45fd" Nov 03 21:27:33.412060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.412027 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l"] Nov 03 21:27:33.412598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.412488 5815 cpu_manager.go:414] "containerMap: removing container" podUID="491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" containerName="revoke-keys" Nov 03 21:27:33.412598 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.412501 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" containerName="revoke-keys" Nov 03 21:27:33.412689 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.412605 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="491c4db8-a01b-4fb4-ba4f-ed5355c17b2c" containerName="revoke-keys" Nov 03 21:27:33.416140 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.416117 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.422837 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.422807 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:27:33.423029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.422941 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:27:33.426637 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.426610 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l"] Nov 03 21:27:33.434018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.433988 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.434171 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.434061 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2rjr\" (UniqueName: \"kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.434171 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.434099 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.535296 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.535255 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2rjr\" (UniqueName: \"kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.535486 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.535307 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.535486 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.535429 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.536101 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.536076 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.537781 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.537761 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.549198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.549176 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2rjr\" (UniqueName: \"kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr\") pod \"maas-api-revoke-sub-602e977b-r2d2l\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:33.731712 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:33.731689 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:34.008184 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:34.008162 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l"] Nov 03 21:27:34.010564 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:27:34.010534 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aae556a_b25d_4776_90ef_03cab781e0e2.slice/crio-f98e95c7cd0952b2cfa93c7c2f1dba537d921a310578fa7ff8f9a10822039437 WatchSource:0}: Error finding container f98e95c7cd0952b2cfa93c7c2f1dba537d921a310578fa7ff8f9a10822039437: Status 404 returned error can't find the container with id f98e95c7cd0952b2cfa93c7c2f1dba537d921a310578fa7ff8f9a10822039437 Nov 03 21:27:34.727221 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:34.727185 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:27:34.815560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:34.815528 5815 generic.go:356] "Generic (PLEG): container finished" podID="3aae556a-b25d-4776-90ef-03cab781e0e2" containerID="d21c68e30ae0b77bb9ffc7067cc8c5676c73066e4c6be6b0433e8cd844a72279" exitCode=0 Nov 03 21:27:34.815667 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:34.815550 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" event={"ID":"3aae556a-b25d-4776-90ef-03cab781e0e2","Type":"ContainerDied","Data":"d21c68e30ae0b77bb9ffc7067cc8c5676c73066e4c6be6b0433e8cd844a72279"} Nov 03 21:27:34.815667 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:34.815611 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" event={"ID":"3aae556a-b25d-4776-90ef-03cab781e0e2","Type":"ContainerStarted","Data":"f98e95c7cd0952b2cfa93c7c2f1dba537d921a310578fa7ff8f9a10822039437"} Nov 03 21:27:35.977104 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:35.977067 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:36.058942 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.058910 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca\") pod \"3aae556a-b25d-4776-90ef-03cab781e0e2\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " Nov 03 21:27:36.059119 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.059074 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca\") pod \"3aae556a-b25d-4776-90ef-03cab781e0e2\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " Nov 03 21:27:36.059178 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.059146 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr\" (UniqueName: \"kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr\") pod \"3aae556a-b25d-4776-90ef-03cab781e0e2\" (UID: \"3aae556a-b25d-4776-90ef-03cab781e0e2\") " Nov 03 21:27:36.059553 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.059486 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca" pod "3aae556a-b25d-4776-90ef-03cab781e0e2" (UID: "3aae556a-b25d-4776-90ef-03cab781e0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:36.061303 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.061276 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca" pod "3aae556a-b25d-4776-90ef-03cab781e0e2" (UID: "3aae556a-b25d-4776-90ef-03cab781e0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:36.062011 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.061985 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr" pod "3aae556a-b25d-4776-90ef-03cab781e0e2" (UID: "3aae556a-b25d-4776-90ef-03cab781e0e2"). InnerVolumeSpecName "kube-api-access-h2rjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:36.161065 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.161005 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:36.161065 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.161041 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2rjr\" (UniqueName: \"kubernetes.io/projected/3aae556a-b25d-4776-90ef-03cab781e0e2-kube-api-access-h2rjr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:36.161065 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.161059 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3aae556a-b25d-4776-90ef-03cab781e0e2-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:36.838727 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.838692 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" event={"ID":"3aae556a-b25d-4776-90ef-03cab781e0e2","Type":"ContainerDied","Data":"f98e95c7cd0952b2cfa93c7c2f1dba537d921a310578fa7ff8f9a10822039437"} Nov 03 21:27:36.838941 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.838750 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98e95c7cd0952b2cfa93c7c2f1dba537d921a310578fa7ff8f9a10822039437" Nov 03 21:27:36.838941 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:36.838814 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602e977b-r2d2l" Nov 03 21:27:43.616580 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.616545 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz"] Nov 03 21:27:43.617022 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.617000 5815 cpu_manager.go:414] "containerMap: removing container" podUID="3aae556a-b25d-4776-90ef-03cab781e0e2" containerName="revoke-keys" Nov 03 21:27:43.617022 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.617014 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aae556a-b25d-4776-90ef-03cab781e0e2" containerName="revoke-keys" Nov 03 21:27:43.617119 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.617079 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aae556a-b25d-4776-90ef-03cab781e0e2" containerName="revoke-keys" Nov 03 21:27:43.620317 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.620296 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.626175 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.626154 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:27:43.626364 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.626246 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:27:43.630985 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.630955 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz"] Nov 03 21:27:43.687198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.687167 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.687391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.687288 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.687391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.687324 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvrq\" (UniqueName: \"kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.787956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.787925 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.787956 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.787970 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvrq\" (UniqueName: \"kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.788237 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.788020 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.788658 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.788635 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.790354 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.790335 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.800133 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.800111 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvrq\" (UniqueName: \"kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq\") pod \"maas-api-revoke-sub-360b246d-x6bvz\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:43.849312 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:27:43.849282 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:27:43.936006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:43.935938 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:44.226041 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:44.226015 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz"] Nov 03 21:27:44.228272 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:27:44.228247 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e24e5bc_908a_4de4_aeb0_0101c17da458.slice/crio-421dc39a35122325007355dcb621de4e2adbbdcb2c94bd41720a79dda0ac78b5 WatchSource:0}: Error finding container 421dc39a35122325007355dcb621de4e2adbbdcb2c94bd41720a79dda0ac78b5: Status 404 returned error can't find the container with id 421dc39a35122325007355dcb621de4e2adbbdcb2c94bd41720a79dda0ac78b5 Nov 03 21:27:44.889431 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:44.889345 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" event={"ID":"8e24e5bc-908a-4de4-aeb0-0101c17da458","Type":"ContainerStarted","Data":"714db92e58b09b0c1d784ac365bfc690f5ace111793ae8c128a944f67e6b366e"} Nov 03 21:27:44.889837 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:44.889448 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" event={"ID":"8e24e5bc-908a-4de4-aeb0-0101c17da458","Type":"ContainerStarted","Data":"421dc39a35122325007355dcb621de4e2adbbdcb2c94bd41720a79dda0ac78b5"} Nov 03 21:27:44.910153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:44.910106 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" podStartSLOduration=1.9100942349999999 podStartE2EDuration="1.910094235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:44.907298472 +0000 UTC m=+5174.437644147" watchObservedRunningTime="2026-11-03 21:27:44.910094235 +0000 UTC m=+5174.440439924" Nov 03 21:27:45.897371 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:45.897333 5815 generic.go:356] "Generic (PLEG): container finished" podID="8e24e5bc-908a-4de4-aeb0-0101c17da458" containerID="714db92e58b09b0c1d784ac365bfc690f5ace111793ae8c128a944f67e6b366e" exitCode=0 Nov 03 21:27:45.897839 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:45.897413 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" event={"ID":"8e24e5bc-908a-4de4-aeb0-0101c17da458","Type":"ContainerDied","Data":"714db92e58b09b0c1d784ac365bfc690f5ace111793ae8c128a944f67e6b366e"} Nov 03 21:27:47.041957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.041933 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:47.119320 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.119289 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca\") pod \"8e24e5bc-908a-4de4-aeb0-0101c17da458\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " Nov 03 21:27:47.119498 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.119330 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca\") pod \"8e24e5bc-908a-4de4-aeb0-0101c17da458\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " Nov 03 21:27:47.119498 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.119378 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq\" (UniqueName: \"kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq\") pod \"8e24e5bc-908a-4de4-aeb0-0101c17da458\" (UID: \"8e24e5bc-908a-4de4-aeb0-0101c17da458\") " Nov 03 21:27:47.119736 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.119713 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca" pod "8e24e5bc-908a-4de4-aeb0-0101c17da458" (UID: "8e24e5bc-908a-4de4-aeb0-0101c17da458"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:47.121704 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.121671 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca" pod "8e24e5bc-908a-4de4-aeb0-0101c17da458" (UID: "8e24e5bc-908a-4de4-aeb0-0101c17da458"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:47.121816 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.121749 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq" pod "8e24e5bc-908a-4de4-aeb0-0101c17da458" (UID: "8e24e5bc-908a-4de4-aeb0-0101c17da458"). InnerVolumeSpecName "kube-api-access-4gvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:47.220352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.220330 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:47.220463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.220354 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e24e5bc-908a-4de4-aeb0-0101c17da458-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:47.220463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.220366 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvrq\" (UniqueName: \"kubernetes.io/projected/8e24e5bc-908a-4de4-aeb0-0101c17da458-kube-api-access-4gvrq\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:47.910113 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.910080 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" event={"ID":"8e24e5bc-908a-4de4-aeb0-0101c17da458","Type":"ContainerDied","Data":"421dc39a35122325007355dcb621de4e2adbbdcb2c94bd41720a79dda0ac78b5"} Nov 03 21:27:47.910113 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.910096 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360b246d-x6bvz" Nov 03 21:27:47.910345 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:47.910126 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421dc39a35122325007355dcb621de4e2adbbdcb2c94bd41720a79dda0ac78b5" Nov 03 21:27:53.833623 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.833592 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz"] Nov 03 21:27:53.834039 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.833976 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8e24e5bc-908a-4de4-aeb0-0101c17da458" containerName="revoke-keys" Nov 03 21:27:53.834039 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.833987 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e24e5bc-908a-4de4-aeb0-0101c17da458" containerName="revoke-keys" Nov 03 21:27:53.834141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.834087 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e24e5bc-908a-4de4-aeb0-0101c17da458" containerName="revoke-keys" Nov 03 21:27:53.837680 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.837654 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:53.843789 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.843766 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:27:53.843983 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.843894 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:27:53.846294 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.846239 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz"] Nov 03 21:27:53.979187 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.979155 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:53.979357 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.979208 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:53.979357 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:53.979307 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rkf\" (UniqueName: \"kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.080715 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.080684 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.080875 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.080736 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.080875 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.080774 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rkf\" (UniqueName: \"kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.081308 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.081289 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.083326 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.083304 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.093535 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.093459 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rkf\" (UniqueName: \"kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf\") pod \"maas-api-revoke-sub-1a3ccb15-g6xwz\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.152953 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.152933 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:54.445171 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.445149 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz"] Nov 03 21:27:54.447383 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:27:54.447350 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936ee3e7_5894_4346_b389_6ce334512f46.slice/crio-f4e6a0a1ccde43b3509487185a000edd28ee291c31b65b06c0e2586e44a8197d WatchSource:0}: Error finding container f4e6a0a1ccde43b3509487185a000edd28ee291c31b65b06c0e2586e44a8197d: Status 404 returned error can't find the container with id f4e6a0a1ccde43b3509487185a000edd28ee291c31b65b06c0e2586e44a8197d Nov 03 21:27:54.962478 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.962443 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" event={"ID":"936ee3e7-5894-4346-b389-6ce334512f46","Type":"ContainerStarted","Data":"eb308ed60c1e22acdce95fe4fc38ae661b5b663a8ab9df01bda944464b02c22e"} Nov 03 21:27:54.962907 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.962496 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" event={"ID":"936ee3e7-5894-4346-b389-6ce334512f46","Type":"ContainerStarted","Data":"f4e6a0a1ccde43b3509487185a000edd28ee291c31b65b06c0e2586e44a8197d"} Nov 03 21:27:54.982208 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:54.982156 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" podStartSLOduration=1.9821429940000002 podStartE2EDuration="1.982142994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:54.980375111 +0000 UTC m=+5184.510720780" watchObservedRunningTime="2026-11-03 21:27:54.982142994 +0000 UTC m=+5184.512488693" Nov 03 21:27:55.970718 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:55.970675 5815 generic.go:356] "Generic (PLEG): container finished" podID="936ee3e7-5894-4346-b389-6ce334512f46" containerID="eb308ed60c1e22acdce95fe4fc38ae661b5b663a8ab9df01bda944464b02c22e" exitCode=0 Nov 03 21:27:55.971235 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:55.970767 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" event={"ID":"936ee3e7-5894-4346-b389-6ce334512f46","Type":"ContainerDied","Data":"eb308ed60c1e22acdce95fe4fc38ae661b5b663a8ab9df01bda944464b02c22e"} Nov 03 21:27:57.112579 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.112553 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:57.212220 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.212192 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca\") pod \"936ee3e7-5894-4346-b389-6ce334512f46\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " Nov 03 21:27:57.212378 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.212280 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca\") pod \"936ee3e7-5894-4346-b389-6ce334512f46\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " Nov 03 21:27:57.212378 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.212316 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf\" (UniqueName: \"kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf\") pod \"936ee3e7-5894-4346-b389-6ce334512f46\" (UID: \"936ee3e7-5894-4346-b389-6ce334512f46\") " Nov 03 21:27:57.212703 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.212677 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca" pod "936ee3e7-5894-4346-b389-6ce334512f46" (UID: "936ee3e7-5894-4346-b389-6ce334512f46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:57.214610 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.214577 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca" pod "936ee3e7-5894-4346-b389-6ce334512f46" (UID: "936ee3e7-5894-4346-b389-6ce334512f46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:57.214723 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.214615 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf" pod "936ee3e7-5894-4346-b389-6ce334512f46" (UID: "936ee3e7-5894-4346-b389-6ce334512f46"). InnerVolumeSpecName "kube-api-access-49rkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:57.313656 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.313603 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:57.313656 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.313624 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936ee3e7-5894-4346-b389-6ce334512f46-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:57.313656 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.313635 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rkf\" (UniqueName: \"kubernetes.io/projected/936ee3e7-5894-4346-b389-6ce334512f46-kube-api-access-49rkf\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:57.982551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.982520 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" event={"ID":"936ee3e7-5894-4346-b389-6ce334512f46","Type":"ContainerDied","Data":"f4e6a0a1ccde43b3509487185a000edd28ee291c31b65b06c0e2586e44a8197d"} Nov 03 21:27:57.982734 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.982556 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3ccb15-g6xwz" Nov 03 21:27:57.982734 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:27:57.982566 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e6a0a1ccde43b3509487185a000edd28ee291c31b65b06c0e2586e44a8197d" Nov 03 21:28:04.044073 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.044043 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx"] Nov 03 21:28:04.044528 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.044454 5815 cpu_manager.go:414] "containerMap: removing container" podUID="936ee3e7-5894-4346-b389-6ce334512f46" containerName="revoke-keys" Nov 03 21:28:04.044528 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.044465 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936ee3e7-5894-4346-b389-6ce334512f46" containerName="revoke-keys" Nov 03 21:28:04.044628 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.044590 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="936ee3e7-5894-4346-b389-6ce334512f46" containerName="revoke-keys" Nov 03 21:28:04.047813 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.047786 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.054022 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.054002 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:28:04.054180 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.054058 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:04.056859 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.056839 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx"] Nov 03 21:28:04.074909 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.074876 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.075059 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.074923 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.075163 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.075078 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldtx\" (UniqueName: \"kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.176243 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.176211 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldtx\" (UniqueName: \"kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.176391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.176273 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.176391 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.176305 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.176967 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.176945 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.178848 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.178830 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.188569 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.188549 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldtx\" (UniqueName: \"kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx\") pod \"maas-api-revoke-sub-8c360db8-zkgrx\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.362864 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.362805 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:04.640029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:04.640002 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx"] Nov 03 21:28:04.642740 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:28:04.642712 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd56ff8e_2eea_4c17_8388_9d25a9e3c7d2.slice/crio-ab1b6b871c267aa404ff842152bb32628d15577bf48807bbc68de67fb1e24d8e WatchSource:0}: Error finding container ab1b6b871c267aa404ff842152bb32628d15577bf48807bbc68de67fb1e24d8e: Status 404 returned error can't find the container with id ab1b6b871c267aa404ff842152bb32628d15577bf48807bbc68de67fb1e24d8e Nov 03 21:28:05.022481 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:05.022455 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" event={"ID":"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2","Type":"ContainerStarted","Data":"e654154cedf8d760c44d0a22079d1cec6d0008ec7743843f8c92688086626d2f"} Nov 03 21:28:05.022642 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:05.022528 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" event={"ID":"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2","Type":"ContainerStarted","Data":"ab1b6b871c267aa404ff842152bb32628d15577bf48807bbc68de67fb1e24d8e"} Nov 03 21:28:05.042000 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:05.041960 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" podStartSLOduration=1.04194662 podStartE2EDuration="1.04194662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:05.039833857 +0000 UTC m=+5194.570179529" watchObservedRunningTime="2026-11-03 21:28:05.04194662 +0000 UTC m=+5194.572292330" Nov 03 21:28:06.033610 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:06.033569 5815 generic.go:356] "Generic (PLEG): container finished" podID="bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" containerID="e654154cedf8d760c44d0a22079d1cec6d0008ec7743843f8c92688086626d2f" exitCode=0 Nov 03 21:28:06.034066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:06.033662 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" event={"ID":"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2","Type":"ContainerDied","Data":"e654154cedf8d760c44d0a22079d1cec6d0008ec7743843f8c92688086626d2f"} Nov 03 21:28:07.439464 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.439434 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:07.553050 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.552979 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca\") pod \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " Nov 03 21:28:07.553050 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.553029 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca\") pod \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " Nov 03 21:28:07.553337 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.553067 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx\" (UniqueName: \"kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx\") pod \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\" (UID: \"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2\") " Nov 03 21:28:07.553865 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.553836 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca" pod "bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" (UID: "bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:07.555715 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.555688 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx" pod "bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" (UID: "bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2"). InnerVolumeSpecName "kube-api-access-sldtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:07.555832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.555748 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca" pod "bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" (UID: "bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:07.654343 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.654266 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sldtx\" (UniqueName: \"kubernetes.io/projected/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-kube-api-access-sldtx\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:07.654343 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.654295 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:07.654343 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:07.654304 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:08.051832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:08.051806 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" Nov 03 21:28:08.052017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:08.051851 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c360db8-zkgrx" event={"ID":"bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2","Type":"ContainerDied","Data":"ab1b6b871c267aa404ff842152bb32628d15577bf48807bbc68de67fb1e24d8e"} Nov 03 21:28:08.052017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:08.051900 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab1b6b871c267aa404ff842152bb32628d15577bf48807bbc68de67fb1e24d8e" Nov 03 21:28:13.850575 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:28:13.850494 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:28:14.262342 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.262315 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd"] Nov 03 21:28:14.262737 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.262724 5815 cpu_manager.go:414] "containerMap: removing container" podUID="bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" containerName="revoke-keys" Nov 03 21:28:14.262796 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.262738 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" containerName="revoke-keys" Nov 03 21:28:14.262843 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.262830 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2" containerName="revoke-keys" Nov 03 21:28:14.265961 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.265940 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.273031 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.272994 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:14.273233 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.273073 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:28:14.275896 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.275474 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd"] Nov 03 21:28:14.312683 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.312657 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64ck\" (UniqueName: \"kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.312857 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.312719 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.312857 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.312822 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.413559 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.413529 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.413746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.413569 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.413746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.413638 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64ck\" (UniqueName: \"kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.414290 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.414264 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.416012 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.415994 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.425974 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.425944 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64ck\" (UniqueName: \"kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck\") pod \"maas-api-revoke-sub-dddb2165-r6csd\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.582900 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.582844 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:14.862546 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:14.862500 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd"] Nov 03 21:28:14.864082 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:28:14.864055 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d98d0e_c64f_47d8_bf56_af9849fc4fb7.slice/crio-2231d953117f82bc3cb1de03a5b8ecf5e01857be7f0a0e456fb8cab8633f92f4 WatchSource:0}: Error finding container 2231d953117f82bc3cb1de03a5b8ecf5e01857be7f0a0e456fb8cab8633f92f4: Status 404 returned error can't find the container with id 2231d953117f82bc3cb1de03a5b8ecf5e01857be7f0a0e456fb8cab8633f92f4 Nov 03 21:28:15.097216 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:15.097187 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" event={"ID":"45d98d0e-c64f-47d8-bf56-af9849fc4fb7","Type":"ContainerStarted","Data":"2231d953117f82bc3cb1de03a5b8ecf5e01857be7f0a0e456fb8cab8633f92f4"} Nov 03 21:28:16.107904 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:16.107867 5815 generic.go:356] "Generic (PLEG): container finished" podID="45d98d0e-c64f-47d8-bf56-af9849fc4fb7" containerID="7f456a6726f987d88dd00330de4a3f4f32cd2ce7690bfc5ab5c2ce155e242ea8" exitCode=0 Nov 03 21:28:16.108346 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:16.107960 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" event={"ID":"45d98d0e-c64f-47d8-bf56-af9849fc4fb7","Type":"ContainerDied","Data":"7f456a6726f987d88dd00330de4a3f4f32cd2ce7690bfc5ab5c2ce155e242ea8"} Nov 03 21:28:17.463745 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.463722 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:17.559950 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.559922 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck\" (UniqueName: \"kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck\") pod \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " Nov 03 21:28:17.560107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.559996 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca\") pod \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " Nov 03 21:28:17.560153 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.560102 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca\") pod \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\" (UID: \"45d98d0e-c64f-47d8-bf56-af9849fc4fb7\") " Nov 03 21:28:17.560689 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.560660 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca" pod "45d98d0e-c64f-47d8-bf56-af9849fc4fb7" (UID: "45d98d0e-c64f-47d8-bf56-af9849fc4fb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:17.562253 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.562225 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck" pod "45d98d0e-c64f-47d8-bf56-af9849fc4fb7" (UID: "45d98d0e-c64f-47d8-bf56-af9849fc4fb7"). InnerVolumeSpecName "kube-api-access-b64ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:17.562449 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.562428 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca" pod "45d98d0e-c64f-47d8-bf56-af9849fc4fb7" (UID: "45d98d0e-c64f-47d8-bf56-af9849fc4fb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:17.661755 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.661724 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:17.661755 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.661751 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64ck\" (UniqueName: \"kubernetes.io/projected/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-kube-api-access-b64ck\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:17.661937 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:17.661761 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45d98d0e-c64f-47d8-bf56-af9849fc4fb7-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:18.144037 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:18.143994 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" event={"ID":"45d98d0e-c64f-47d8-bf56-af9849fc4fb7","Type":"ContainerDied","Data":"2231d953117f82bc3cb1de03a5b8ecf5e01857be7f0a0e456fb8cab8633f92f4"} Nov 03 21:28:18.144228 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:18.144068 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2231d953117f82bc3cb1de03a5b8ecf5e01857be7f0a0e456fb8cab8633f92f4" Nov 03 21:28:18.144228 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:18.144010 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dddb2165-r6csd" Nov 03 21:28:24.480201 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.480165 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h"] Nov 03 21:28:24.480685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.480657 5815 cpu_manager.go:414] "containerMap: removing container" podUID="45d98d0e-c64f-47d8-bf56-af9849fc4fb7" containerName="revoke-keys" Nov 03 21:28:24.480685 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.480673 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d98d0e-c64f-47d8-bf56-af9849fc4fb7" containerName="revoke-keys" Nov 03 21:28:24.480785 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.480749 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d98d0e-c64f-47d8-bf56-af9849fc4fb7" containerName="revoke-keys" Nov 03 21:28:24.484212 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.484196 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.490212 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.490185 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:24.490414 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.490396 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:28:24.494104 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.493062 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h"] Nov 03 21:28:24.527409 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.527383 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4nt\" (UniqueName: \"kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.527562 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.527419 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.527640 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.527615 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.628917 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.628888 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4nt\" (UniqueName: \"kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.629084 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.628929 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.629084 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.629002 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.629676 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.629654 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.631401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.631380 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.641332 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.641309 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4nt\" (UniqueName: \"kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt\") pod \"maas-api-revoke-sub-a54c94f0-99n9h\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:24.803715 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:24.803661 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:25.079623 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:25.079593 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h"] Nov 03 21:28:25.081001 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:28:25.080972 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0470f896_2d1c_4a5b_9a43_48998e396d7e.slice/crio-fb5fb6dd6ec0cd903c532cf7e49941031d15662ff11abc6289b3ca21ec2fd6d8 WatchSource:0}: Error finding container fb5fb6dd6ec0cd903c532cf7e49941031d15662ff11abc6289b3ca21ec2fd6d8: Status 404 returned error can't find the container with id fb5fb6dd6ec0cd903c532cf7e49941031d15662ff11abc6289b3ca21ec2fd6d8 Nov 03 21:28:25.184812 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:25.184785 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" event={"ID":"0470f896-2d1c-4a5b-9a43-48998e396d7e","Type":"ContainerStarted","Data":"fb5fb6dd6ec0cd903c532cf7e49941031d15662ff11abc6289b3ca21ec2fd6d8"} Nov 03 21:28:26.195560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:26.195448 5815 generic.go:356] "Generic (PLEG): container finished" podID="0470f896-2d1c-4a5b-9a43-48998e396d7e" containerID="254417bdae24705bcb3442109d60519a0e2bc4186bb26abb1d8018550331abbf" exitCode=0 Nov 03 21:28:26.195560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:26.195525 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" event={"ID":"0470f896-2d1c-4a5b-9a43-48998e396d7e","Type":"ContainerDied","Data":"254417bdae24705bcb3442109d60519a0e2bc4186bb26abb1d8018550331abbf"} Nov 03 21:28:27.555873 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.555850 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:27.656919 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.656887 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca\") pod \"0470f896-2d1c-4a5b-9a43-48998e396d7e\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " Nov 03 21:28:27.657111 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.656946 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt\" (UniqueName: \"kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt\") pod \"0470f896-2d1c-4a5b-9a43-48998e396d7e\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " Nov 03 21:28:27.657111 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.656997 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca\") pod \"0470f896-2d1c-4a5b-9a43-48998e396d7e\" (UID: \"0470f896-2d1c-4a5b-9a43-48998e396d7e\") " Nov 03 21:28:27.657250 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.657229 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca" pod "0470f896-2d1c-4a5b-9a43-48998e396d7e" (UID: "0470f896-2d1c-4a5b-9a43-48998e396d7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:27.657357 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.657340 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:27.659320 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.659295 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca" pod "0470f896-2d1c-4a5b-9a43-48998e396d7e" (UID: "0470f896-2d1c-4a5b-9a43-48998e396d7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:27.659456 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.659433 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt" pod "0470f896-2d1c-4a5b-9a43-48998e396d7e" (UID: "0470f896-2d1c-4a5b-9a43-48998e396d7e"). InnerVolumeSpecName "kube-api-access-gq4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:27.758062 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.757977 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4nt\" (UniqueName: \"kubernetes.io/projected/0470f896-2d1c-4a5b-9a43-48998e396d7e-kube-api-access-gq4nt\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:27.758062 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:27.758009 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0470f896-2d1c-4a5b-9a43-48998e396d7e-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:28.237172 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:28.237146 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" Nov 03 21:28:28.237565 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:28.237191 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a54c94f0-99n9h" event={"ID":"0470f896-2d1c-4a5b-9a43-48998e396d7e","Type":"ContainerDied","Data":"fb5fb6dd6ec0cd903c532cf7e49941031d15662ff11abc6289b3ca21ec2fd6d8"} Nov 03 21:28:28.237923 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:28.237904 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5fb6dd6ec0cd903c532cf7e49941031d15662ff11abc6289b3ca21ec2fd6d8" Nov 03 21:28:34.678850 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.678818 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd"] Nov 03 21:28:34.679269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.679206 5815 cpu_manager.go:414] "containerMap: removing container" podUID="0470f896-2d1c-4a5b-9a43-48998e396d7e" containerName="revoke-keys" Nov 03 21:28:34.679269 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.679217 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0470f896-2d1c-4a5b-9a43-48998e396d7e" containerName="revoke-keys" Nov 03 21:28:34.679353 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.679289 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="0470f896-2d1c-4a5b-9a43-48998e396d7e" containerName="revoke-keys" Nov 03 21:28:34.682641 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.682624 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.688942 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.688911 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:34.689099 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.689078 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:28:34.691441 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.691419 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd"] Nov 03 21:28:34.716095 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.716056 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.716254 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.716183 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpv8\" (UniqueName: \"kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.716254 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.716247 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.817383 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.817353 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpv8\" (UniqueName: \"kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.817560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.817393 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.817560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.817423 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.818107 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.818077 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.820202 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.820172 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.830241 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.830215 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpv8\" (UniqueName: \"kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8\") pod \"maas-api-revoke-sub-049165b8-8gpfd\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:34.998086 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:34.998050 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:35.276551 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:35.276521 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd"] Nov 03 21:28:35.276907 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:28:35.276888 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b89f655_c6cf_4f99_8979_3cefc55f2be7.slice/crio-07a9a8e26e1fb32ac43f324979e3acef3a984fe1ef9318a595791310902efb25 WatchSource:0}: Error finding container 07a9a8e26e1fb32ac43f324979e3acef3a984fe1ef9318a595791310902efb25: Status 404 returned error can't find the container with id 07a9a8e26e1fb32ac43f324979e3acef3a984fe1ef9318a595791310902efb25 Nov 03 21:28:36.289038 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:36.288995 5815 generic.go:356] "Generic (PLEG): container finished" podID="8b89f655-c6cf-4f99-8979-3cefc55f2be7" containerID="32d7f972e11c2b7a70f05dc7233ce9895b4dc9e09ec06c832df6405cb0cfa2a7" exitCode=0 Nov 03 21:28:36.289557 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:36.289043 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" event={"ID":"8b89f655-c6cf-4f99-8979-3cefc55f2be7","Type":"ContainerDied","Data":"32d7f972e11c2b7a70f05dc7233ce9895b4dc9e09ec06c832df6405cb0cfa2a7"} Nov 03 21:28:36.289557 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:36.289094 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" event={"ID":"8b89f655-c6cf-4f99-8979-3cefc55f2be7","Type":"ContainerStarted","Data":"07a9a8e26e1fb32ac43f324979e3acef3a984fe1ef9318a595791310902efb25"} Nov 03 21:28:37.665943 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.665604 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:37.746726 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.746053 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8\" (UniqueName: \"kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8\") pod \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " Nov 03 21:28:37.746726 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.746139 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca\") pod \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " Nov 03 21:28:37.746726 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.746193 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca\") pod \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\" (UID: \"8b89f655-c6cf-4f99-8979-3cefc55f2be7\") " Nov 03 21:28:37.747617 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.747565 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca" pod "8b89f655-c6cf-4f99-8979-3cefc55f2be7" (UID: "8b89f655-c6cf-4f99-8979-3cefc55f2be7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:37.750933 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.750911 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8" pod "8b89f655-c6cf-4f99-8979-3cefc55f2be7" (UID: "8b89f655-c6cf-4f99-8979-3cefc55f2be7"). InnerVolumeSpecName "kube-api-access-gmpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:37.751359 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.751333 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca" pod "8b89f655-c6cf-4f99-8979-3cefc55f2be7" (UID: "8b89f655-c6cf-4f99-8979-3cefc55f2be7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:37.847906 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.847874 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpv8\" (UniqueName: \"kubernetes.io/projected/8b89f655-c6cf-4f99-8979-3cefc55f2be7-kube-api-access-gmpv8\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:37.847906 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.847905 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:37.847906 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:37.847914 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b89f655-c6cf-4f99-8979-3cefc55f2be7-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:38.341532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:38.341127 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" event={"ID":"8b89f655-c6cf-4f99-8979-3cefc55f2be7","Type":"ContainerDied","Data":"07a9a8e26e1fb32ac43f324979e3acef3a984fe1ef9318a595791310902efb25"} Nov 03 21:28:38.341532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:38.341193 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a9a8e26e1fb32ac43f324979e3acef3a984fe1ef9318a595791310902efb25" Nov 03 21:28:38.341532 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:38.341267 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-049165b8-8gpfd" Nov 03 21:28:43.849479 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:28:43.849424 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:28:44.882884 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.882854 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6"] Nov 03 21:28:44.885289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.883244 5815 cpu_manager.go:414] "containerMap: removing container" podUID="8b89f655-c6cf-4f99-8979-3cefc55f2be7" containerName="revoke-keys" Nov 03 21:28:44.885289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.883254 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b89f655-c6cf-4f99-8979-3cefc55f2be7" containerName="revoke-keys" Nov 03 21:28:44.885289 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.883338 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b89f655-c6cf-4f99-8979-3cefc55f2be7" containerName="revoke-keys" Nov 03 21:28:44.886416 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.886393 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:44.895563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.892693 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:44.895563 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.892876 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:28:44.896195 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.895760 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6"] Nov 03 21:28:44.908460 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.908438 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:44.908645 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.908588 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhvz\" (UniqueName: \"kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:44.908645 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:44.908610 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.009286 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.009256 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhvz\" (UniqueName: \"kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.009286 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.009286 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.009617 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.009312 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.009928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.009899 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.011831 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.011807 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.021613 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.021591 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhvz\" (UniqueName: \"kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz\") pod \"maas-api-revoke-sub-f2fdb378-rrqh6\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.202165 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.202110 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:45.478871 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:45.478844 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6"] Nov 03 21:28:45.480486 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:28:45.480449 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507d64f3_00df_402c_9fe0_637b8688005a.slice/crio-3fbb26f1516bce2355e274fe3a057e087b9d22830cb490f827fc282348121755 WatchSource:0}: Error finding container 3fbb26f1516bce2355e274fe3a057e087b9d22830cb490f827fc282348121755: Status 404 returned error can't find the container with id 3fbb26f1516bce2355e274fe3a057e087b9d22830cb490f827fc282348121755 Nov 03 21:28:46.394928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:46.394890 5815 generic.go:356] "Generic (PLEG): container finished" podID="507d64f3-00df-402c-9fe0-637b8688005a" containerID="a9142b41ee7cc6f0ddeed08e978dc7ff7349b5984ce19bcb3564e7ca8c7f7a14" exitCode=0 Nov 03 21:28:46.395281 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:46.394985 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" event={"ID":"507d64f3-00df-402c-9fe0-637b8688005a","Type":"ContainerDied","Data":"a9142b41ee7cc6f0ddeed08e978dc7ff7349b5984ce19bcb3564e7ca8c7f7a14"} Nov 03 21:28:46.395281 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:46.395084 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" event={"ID":"507d64f3-00df-402c-9fe0-637b8688005a","Type":"ContainerStarted","Data":"3fbb26f1516bce2355e274fe3a057e087b9d22830cb490f827fc282348121755"} Nov 03 21:28:47.800533 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.765451 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:47.847745 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.847714 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz\" (UniqueName: \"kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz\") pod \"507d64f3-00df-402c-9fe0-637b8688005a\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " Nov 03 21:28:47.847973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.847946 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca\") pod \"507d64f3-00df-402c-9fe0-637b8688005a\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " Nov 03 21:28:47.848072 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.848053 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca\") pod \"507d64f3-00df-402c-9fe0-637b8688005a\" (UID: \"507d64f3-00df-402c-9fe0-637b8688005a\") " Nov 03 21:28:47.848646 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.848589 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca" pod "507d64f3-00df-402c-9fe0-637b8688005a" (UID: "507d64f3-00df-402c-9fe0-637b8688005a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:47.851017 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.850990 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz" pod "507d64f3-00df-402c-9fe0-637b8688005a" (UID: "507d64f3-00df-402c-9fe0-637b8688005a"). InnerVolumeSpecName "kube-api-access-8lhvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:47.851355 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.851328 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca" pod "507d64f3-00df-402c-9fe0-637b8688005a" (UID: "507d64f3-00df-402c-9fe0-637b8688005a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:47.949097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.949059 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:47.949097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.949090 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhvz\" (UniqueName: \"kubernetes.io/projected/507d64f3-00df-402c-9fe0-637b8688005a-kube-api-access-8lhvz\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:47.949097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:47.949101 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507d64f3-00df-402c-9fe0-637b8688005a-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:48.445880 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:48.445857 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" Nov 03 21:28:48.446442 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:48.445896 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2fdb378-rrqh6" event={"ID":"507d64f3-00df-402c-9fe0-637b8688005a","Type":"ContainerDied","Data":"3fbb26f1516bce2355e274fe3a057e087b9d22830cb490f827fc282348121755"} Nov 03 21:28:48.446610 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:48.446469 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fbb26f1516bce2355e274fe3a057e087b9d22830cb490f827fc282348121755" Nov 03 21:28:55.094401 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.094367 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv"] Nov 03 21:28:55.094826 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.094775 5815 cpu_manager.go:414] "containerMap: removing container" podUID="507d64f3-00df-402c-9fe0-637b8688005a" containerName="revoke-keys" Nov 03 21:28:55.094826 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.094787 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507d64f3-00df-402c-9fe0-637b8688005a" containerName="revoke-keys" Nov 03 21:28:55.094912 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.094880 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="507d64f3-00df-402c-9fe0-637b8688005a" containerName="revoke-keys" Nov 03 21:28:55.098234 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.098214 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.104418 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.104392 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:55.104578 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.104440 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:28:55.107306 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.107280 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv"] Nov 03 21:28:55.213417 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.213379 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.213606 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.213524 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x6hr\" (UniqueName: \"kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.213606 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.213575 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.314446 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.314415 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.314635 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.314549 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.314635 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.314617 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6hr\" (UniqueName: \"kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.315275 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.315251 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.317142 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.317112 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.327086 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.327061 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6hr\" (UniqueName: \"kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr\") pod \"maas-api-revoke-sub-1c9726d7-vvrsv\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.413701 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.413647 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:55.693456 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:55.693413 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv"] Nov 03 21:28:55.694986 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:28:55.694953 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c78690e_6553_47b9_92eb_569b2e9f9a49.slice/crio-a6efa6739cf72124ee1d91d85282e86c208aa9ca7f62eb0ea0c15d0259476562 WatchSource:0}: Error finding container a6efa6739cf72124ee1d91d85282e86c208aa9ca7f62eb0ea0c15d0259476562: Status 404 returned error can't find the container with id a6efa6739cf72124ee1d91d85282e86c208aa9ca7f62eb0ea0c15d0259476562 Nov 03 21:28:56.500373 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:56.500327 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" event={"ID":"4c78690e-6553-47b9-92eb-569b2e9f9a49","Type":"ContainerStarted","Data":"e720286e3e828b701f382e1d25e6c580c21e39db95ae3492d049340610fa6d44"} Nov 03 21:28:56.500812 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:56.500449 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" event={"ID":"4c78690e-6553-47b9-92eb-569b2e9f9a49","Type":"ContainerStarted","Data":"a6efa6739cf72124ee1d91d85282e86c208aa9ca7f62eb0ea0c15d0259476562"} Nov 03 21:28:56.520486 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:56.520434 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" podStartSLOduration=1.520415242 podStartE2EDuration="1.520415242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:56.52035076 +0000 UTC m=+5246.050696437" watchObservedRunningTime="2026-11-03 21:28:56.520415242 +0000 UTC m=+5246.050760915" Nov 03 21:28:57.532064 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:57.532030 5815 generic.go:356] "Generic (PLEG): container finished" podID="4c78690e-6553-47b9-92eb-569b2e9f9a49" containerID="e720286e3e828b701f382e1d25e6c580c21e39db95ae3492d049340610fa6d44" exitCode=0 Nov 03 21:28:57.532853 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:57.532827 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" event={"ID":"4c78690e-6553-47b9-92eb-569b2e9f9a49","Type":"ContainerDied","Data":"e720286e3e828b701f382e1d25e6c580c21e39db95ae3492d049340610fa6d44"} Nov 03 21:28:58.961104 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:58.961080 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:59.054222 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.054190 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca\") pod \"4c78690e-6553-47b9-92eb-569b2e9f9a49\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " Nov 03 21:28:59.054408 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.054225 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca\") pod \"4c78690e-6553-47b9-92eb-569b2e9f9a49\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " Nov 03 21:28:59.054408 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.054284 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr\" (UniqueName: \"kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr\") pod \"4c78690e-6553-47b9-92eb-569b2e9f9a49\" (UID: \"4c78690e-6553-47b9-92eb-569b2e9f9a49\") " Nov 03 21:28:59.054705 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.054669 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca" pod "4c78690e-6553-47b9-92eb-569b2e9f9a49" (UID: "4c78690e-6553-47b9-92eb-569b2e9f9a49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:59.056610 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.056585 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca" pod "4c78690e-6553-47b9-92eb-569b2e9f9a49" (UID: "4c78690e-6553-47b9-92eb-569b2e9f9a49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:59.056760 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.056737 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr" pod "4c78690e-6553-47b9-92eb-569b2e9f9a49" (UID: "4c78690e-6553-47b9-92eb-569b2e9f9a49"). InnerVolumeSpecName "kube-api-access-6x6hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:59.155421 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.155380 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:59.155421 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.155423 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c78690e-6553-47b9-92eb-569b2e9f9a49-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:59.155640 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.155441 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x6hr\" (UniqueName: \"kubernetes.io/projected/4c78690e-6553-47b9-92eb-569b2e9f9a49-kube-api-access-6x6hr\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:59.553961 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.553924 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" event={"ID":"4c78690e-6553-47b9-92eb-569b2e9f9a49","Type":"ContainerDied","Data":"a6efa6739cf72124ee1d91d85282e86c208aa9ca7f62eb0ea0c15d0259476562"} Nov 03 21:28:59.553961 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.553957 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c9726d7-vvrsv" Nov 03 21:28:59.554194 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:28:59.553980 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6efa6739cf72124ee1d91d85282e86c208aa9ca7f62eb0ea0c15d0259476562" Nov 03 21:29:05.300170 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.300137 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4"] Nov 03 21:29:05.300739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.300589 5815 cpu_manager.go:414] "containerMap: removing container" podUID="4c78690e-6553-47b9-92eb-569b2e9f9a49" containerName="revoke-keys" Nov 03 21:29:05.300739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.300602 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c78690e-6553-47b9-92eb-569b2e9f9a49" containerName="revoke-keys" Nov 03 21:29:05.300739 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.300682 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c78690e-6553-47b9-92eb-569b2e9f9a49" containerName="revoke-keys" Nov 03 21:29:05.304145 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.304123 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.309030 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.308987 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:29:05.310488 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.310444 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:29:05.314784 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.314760 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4"] Nov 03 21:29:05.409795 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.409760 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.410005 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.409827 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.410005 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.409940 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9z6\" (UniqueName: \"kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.510887 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.510859 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.511060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.510910 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.511060 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.510937 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9z6\" (UniqueName: \"kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.511536 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.511490 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.513569 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.513546 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.523013 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.522989 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9z6\" (UniqueName: \"kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6\") pod \"maas-api-revoke-sub-84112417-7mjb4\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.619686 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.619638 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:05.897094 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:05.897071 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4"] Nov 03 21:29:05.898861 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:29:05.898830 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e8c38b_d8d8_40ec_8640_20dbcc7270ce.slice/crio-ffe04dd7cdbdbd62691260997240a4177428e34848d69ccc576bfd69da4a24f5 WatchSource:0}: Error finding container ffe04dd7cdbdbd62691260997240a4177428e34848d69ccc576bfd69da4a24f5: Status 404 returned error can't find the container with id ffe04dd7cdbdbd62691260997240a4177428e34848d69ccc576bfd69da4a24f5 Nov 03 21:29:06.599722 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:06.599684 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" event={"ID":"56e8c38b-d8d8-40ec-8640-20dbcc7270ce","Type":"ContainerStarted","Data":"9e82fb55a3f0e2f5aa71a7ee8e28a6901de0540e19620dd74bf108b4799b8c73"} Nov 03 21:29:06.600374 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:06.599733 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" event={"ID":"56e8c38b-d8d8-40ec-8640-20dbcc7270ce","Type":"ContainerStarted","Data":"ffe04dd7cdbdbd62691260997240a4177428e34848d69ccc576bfd69da4a24f5"} Nov 03 21:29:06.620816 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:06.620769 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" podStartSLOduration=1.6207560829999998 podStartE2EDuration="1.620756083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:06.618131317 +0000 UTC m=+5256.148476989" watchObservedRunningTime="2026-11-03 21:29:06.620756083 +0000 UTC m=+5256.151101757" Nov 03 21:29:07.639141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:07.639010 5815 generic.go:356] "Generic (PLEG): container finished" podID="56e8c38b-d8d8-40ec-8640-20dbcc7270ce" containerID="9e82fb55a3f0e2f5aa71a7ee8e28a6901de0540e19620dd74bf108b4799b8c73" exitCode=0 Nov 03 21:29:07.639141 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:07.639075 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" event={"ID":"56e8c38b-d8d8-40ec-8640-20dbcc7270ce","Type":"ContainerDied","Data":"9e82fb55a3f0e2f5aa71a7ee8e28a6901de0540e19620dd74bf108b4799b8c73"} Nov 03 21:29:09.051839 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.051801 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:09.152279 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.152247 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca\") pod \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " Nov 03 21:29:09.152485 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.152292 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca\") pod \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " Nov 03 21:29:09.152485 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.152333 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6\" (UniqueName: \"kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6\") pod \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\" (UID: \"56e8c38b-d8d8-40ec-8640-20dbcc7270ce\") " Nov 03 21:29:09.152723 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.152693 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca" pod "56e8c38b-d8d8-40ec-8640-20dbcc7270ce" (UID: "56e8c38b-d8d8-40ec-8640-20dbcc7270ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:09.155347 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.155318 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca" pod "56e8c38b-d8d8-40ec-8640-20dbcc7270ce" (UID: "56e8c38b-d8d8-40ec-8640-20dbcc7270ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:09.155525 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.155483 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6" pod "56e8c38b-d8d8-40ec-8640-20dbcc7270ce" (UID: "56e8c38b-d8d8-40ec-8640-20dbcc7270ce"). InnerVolumeSpecName "kube-api-access-fd9z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:09.253709 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.253636 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:09.253709 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.253663 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:09.253709 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.253673 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd9z6\" (UniqueName: \"kubernetes.io/projected/56e8c38b-d8d8-40ec-8640-20dbcc7270ce-kube-api-access-fd9z6\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:09.662115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.662030 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" event={"ID":"56e8c38b-d8d8-40ec-8640-20dbcc7270ce","Type":"ContainerDied","Data":"ffe04dd7cdbdbd62691260997240a4177428e34848d69ccc576bfd69da4a24f5"} Nov 03 21:29:09.662115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.662061 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84112417-7mjb4" Nov 03 21:29:09.662115 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:09.662081 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe04dd7cdbdbd62691260997240a4177428e34848d69ccc576bfd69da4a24f5" Nov 03 21:29:13.854034 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:29:13.853998 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:29:15.516058 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.516012 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c"] Nov 03 21:29:15.516926 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.516899 5815 cpu_manager.go:414] "containerMap: removing container" podUID="56e8c38b-d8d8-40ec-8640-20dbcc7270ce" containerName="revoke-keys" Nov 03 21:29:15.516926 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.516928 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e8c38b-d8d8-40ec-8640-20dbcc7270ce" containerName="revoke-keys" Nov 03 21:29:15.517172 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.517154 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e8c38b-d8d8-40ec-8640-20dbcc7270ce" containerName="revoke-keys" Nov 03 21:29:15.521370 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.521326 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.524970 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.524946 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c"] Nov 03 21:29:15.525599 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.525575 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:29:15.526871 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.526848 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:29:15.618965 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.618936 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj45f\" (UniqueName: \"kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.619130 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.619109 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.619197 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.619170 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.719703 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.719669 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj45f\" (UniqueName: \"kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.719865 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.719769 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.719865 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.719806 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.720387 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.720359 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.722371 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.722347 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.732206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.732185 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj45f\" (UniqueName: \"kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f\") pod \"maas-api-revoke-sub-aec795a8-5wd2c\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:15.836544 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:15.836457 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:16.114597 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:16.114567 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c"] Nov 03 21:29:16.115616 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:29:16.115587 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54150e55_baad_46ff_a56d_90ba8c821278.slice/crio-645570066eb2827db9da8806b58f118e10439fc624089ea9837ef03b285dfd12 WatchSource:0}: Error finding container 645570066eb2827db9da8806b58f118e10439fc624089ea9837ef03b285dfd12: Status 404 returned error can't find the container with id 645570066eb2827db9da8806b58f118e10439fc624089ea9837ef03b285dfd12 Nov 03 21:29:16.706746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:16.706653 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" event={"ID":"54150e55-baad-46ff-a56d-90ba8c821278","Type":"ContainerStarted","Data":"8fbf1956fae1bb3f1b7185164ca3e6a4dcfbebacec0c904c6acd020e6bb3a080"} Nov 03 21:29:16.706746 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:16.706716 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" event={"ID":"54150e55-baad-46ff-a56d-90ba8c821278","Type":"ContainerStarted","Data":"645570066eb2827db9da8806b58f118e10439fc624089ea9837ef03b285dfd12"} Nov 03 21:29:16.727674 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:16.727605 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" podStartSLOduration=1.727589944 podStartE2EDuration="1.727589944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:29:16.726447997 +0000 UTC m=+5266.256793672" watchObservedRunningTime="2026-11-03 21:29:16.727589944 +0000 UTC m=+5266.257935618" Nov 03 21:29:17.742002 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:17.741864 5815 generic.go:356] "Generic (PLEG): container finished" podID="54150e55-baad-46ff-a56d-90ba8c821278" containerID="8fbf1956fae1bb3f1b7185164ca3e6a4dcfbebacec0c904c6acd020e6bb3a080" exitCode=0 Nov 03 21:29:17.742002 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:17.741913 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" event={"ID":"54150e55-baad-46ff-a56d-90ba8c821278","Type":"ContainerDied","Data":"8fbf1956fae1bb3f1b7185164ca3e6a4dcfbebacec0c904c6acd020e6bb3a080"} Nov 03 21:29:18.888677 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.888653 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:18.954262 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.954232 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f\" (UniqueName: \"kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f\") pod \"54150e55-baad-46ff-a56d-90ba8c821278\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " Nov 03 21:29:18.954444 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.954355 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca\") pod \"54150e55-baad-46ff-a56d-90ba8c821278\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " Nov 03 21:29:18.954558 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.954464 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca\") pod \"54150e55-baad-46ff-a56d-90ba8c821278\" (UID: \"54150e55-baad-46ff-a56d-90ba8c821278\") " Nov 03 21:29:18.954791 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.954763 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca" pod "54150e55-baad-46ff-a56d-90ba8c821278" (UID: "54150e55-baad-46ff-a56d-90ba8c821278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:18.956617 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.956598 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca" pod "54150e55-baad-46ff-a56d-90ba8c821278" (UID: "54150e55-baad-46ff-a56d-90ba8c821278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:18.956744 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:18.956724 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f" pod "54150e55-baad-46ff-a56d-90ba8c821278" (UID: "54150e55-baad-46ff-a56d-90ba8c821278"). InnerVolumeSpecName "kube-api-access-sj45f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:19.055198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:19.055131 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-xks-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:19.055198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:19.055155 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj45f\" (UniqueName: \"kubernetes.io/projected/54150e55-baad-46ff-a56d-90ba8c821278-kube-api-access-sj45f\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:19.055198 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:19.055164 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54150e55-baad-46ff-a56d-90ba8c821278-maas-api-openshift-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:19.755817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:19.755787 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" Nov 03 21:29:19.756006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:19.755781 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec795a8-5wd2c" event={"ID":"54150e55-baad-46ff-a56d-90ba8c821278","Type":"ContainerDied","Data":"645570066eb2827db9da8806b58f118e10439fc624089ea9837ef03b285dfd12"} Nov 03 21:29:19.756006 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:19.755907 5815 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="645570066eb2827db9da8806b58f118e10439fc624089ea9837ef03b285dfd12" Nov 03 21:29:26.455560 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:26.455526 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-q9gdw"] Nov 03 21:29:43.850282 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:29:43.850241 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:29:48.727403 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:48.727369 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q5w7x_09dae28a-2531-444a-bf0c-bbe040b3b92d/manager/0.log" Nov 03 21:29:50.881282 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:50.881247 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-049165b8-8gpfd_8b89f655-c6cf-4f99-8979-3cefc55f2be7/revoke-keys/0.log" Nov 03 21:29:51.064075 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:51.064044 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1a3ccb15-g6xwz_936ee3e7-5894-4346-b389-6ce334512f46/revoke-keys/0.log" Nov 03 21:29:51.234240 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:51.234216 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c9726d7-vvrsv_4c78690e-6553-47b9-92eb-569b2e9f9a49/revoke-keys/0.log" Nov 03 21:29:51.409229 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:51.409197 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-227e46b3-nj8ls_491c4db8-a01b-4fb4-ba4f-ed5355c17b2c/revoke-keys/0.log" Nov 03 21:29:51.596446 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:51.596357 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2deb6417-r9hs7_c22a9d03-82a9-4d24-b85c-4b600817845a/revoke-keys/0.log" Nov 03 21:29:51.777589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:51.777556 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3400d899-278b9_42972838-3a85-4746-833e-f73b9d3f478e/revoke-keys/0.log" Nov 03 21:29:51.966955 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:51.966931 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-360b246d-x6bvz_8e24e5bc-908a-4de4-aeb0-0101c17da458/revoke-keys/0.log" Nov 03 21:29:52.139252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:52.139223 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3a6e4be7-844p2_3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a/revoke-keys/0.log" Nov 03 21:29:52.312886 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:52.312823 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-602e977b-r2d2l_3aae556a-b25d-4776-90ef-03cab781e0e2/revoke-keys/0.log" Nov 03 21:29:52.482486 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:52.482454 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84112417-7mjb4_56e8c38b-d8d8-40ec-8640-20dbcc7270ce/revoke-keys/0.log" Nov 03 21:29:52.656266 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:52.656200 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84d70be0-gmsnx_a21977a0-57a9-43b7-97be-67a035b14ebd/revoke-keys/0.log" Nov 03 21:29:52.820322 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:52.820298 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8c360db8-zkgrx_bd56ff8e-2eea-4c17-8388-9d25a9e3c7d2/revoke-keys/0.log" Nov 03 21:29:52.997994 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:52.997945 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-91e55dcb-764m8_4df23b8b-3a84-48bc-aad8-fe9961216d50/revoke-keys/0.log" Nov 03 21:29:53.176877 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:53.176849 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a54c94f0-99n9h_0470f896-2d1c-4a5b-9a43-48998e396d7e/revoke-keys/0.log" Nov 03 21:29:53.359867 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:53.359796 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aec795a8-5wd2c_54150e55-baad-46ff-a56d-90ba8c821278/revoke-keys/0.log" Nov 03 21:29:53.529381 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:53.529361 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-daee2067-xjkfl_49444382-aa13-4ee0-8009-28e4f5892c5a/revoke-keys/0.log" Nov 03 21:29:53.712431 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:53.712407 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dddb2165-r6csd_45d98d0e-c64f-47d8-bf56-af9849fc4fb7/revoke-keys/0.log" Nov 03 21:29:53.892473 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:53.892452 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f2fdb378-rrqh6_507d64f3-00df-402c-9fe0-637b8688005a/revoke-keys/0.log" Nov 03 21:29:54.064733 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:54.064671 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f829c667-qzg9q_ccac898a-c58d-4879-8b4f-c78b67c2ff6a/revoke-keys/0.log" Nov 03 21:29:54.240245 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:54.240213 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-bs6vn_e5598e3b-f89f-47d0-a9bb-6099846c5ee2/postgres/0.log" Nov 03 21:29:56.302675 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:56.302644 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-bx9l8_2276a55f-e1a9-406a-994e-ace6113b9d42/manager/0.log" Nov 03 21:29:56.497529 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:56.497486 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jgwq4_df4b5345-2b3d-425d-a9f9-2dc3f382fcf7/manager/0.log" Nov 03 21:29:57.139167 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:57.139136 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-q9gdw_683dd503-5b16-4acc-9d0f-36fd1baf048e/limitador/0.log" Nov 03 21:29:57.325783 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:57.325752 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h946s_9da9ab34-02dc-458e-84d7-e9522d44c516/manager/0.log" Nov 03 21:29:58.275064 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:58.275034 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-76f56c5945zkwwr_e5cb68d0-ef10-4bc6-9677-5eff1054ca0f/istio-proxy/0.log" Nov 03 21:29:58.465825 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:58.465779 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x96lq_b69e4a59-126c-4592-b7f5-b2ccf31781d7/discovery/0.log" Nov 03 21:29:59.024761 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:59.024722 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2"] Nov 03 21:29:59.027446 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:59.027425 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6e4be7-844p2"] Nov 03 21:29:59.770860 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:59.770825 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-gqpt8_ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9/istio-proxy/0.log" Nov 03 21:29:59.967928 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:29:59.967897 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-h8zwm_44953e3d-f118-443e-894d-530fba487260/istio-proxy/0.log" Nov 03 21:30:00.143616 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.143534 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5"] Nov 03 21:30:00.143995 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.143983 5815 cpu_manager.go:414] "containerMap: removing container" podUID="54150e55-baad-46ff-a56d-90ba8c821278" containerName="revoke-keys" Nov 03 21:30:00.144048 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.143997 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54150e55-baad-46ff-a56d-90ba8c821278" containerName="revoke-keys" Nov 03 21:30:00.144091 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.144072 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="54150e55-baad-46ff-a56d-90ba8c821278" containerName="revoke-keys" Nov 03 21:30:00.147654 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.147638 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.156364 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.156340 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:30:00.158984 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.158936 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5"] Nov 03 21:30:00.220405 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.220374 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncv2\" (UniqueName: \"kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2\") pod \"maas-api-key-cleanup-29895690-jt9k5\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.220609 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.220544 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-jt9k5\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.321799 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.321768 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-jt9k5\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.321973 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.321836 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncv2\" (UniqueName: \"kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2\") pod \"maas-api-key-cleanup-29895690-jt9k5\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.322461 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.322441 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895690-jt9k5\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.334987 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.334966 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncv2\" (UniqueName: \"kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2\") pod \"maas-api-key-cleanup-29895690-jt9k5\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.462603 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.462578 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:00.584589 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.584559 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a" path="/var/lib/kubelet/pods/3f1c0986-8eb3-4d0b-9b31-9b785c7fab4a/volumes" Nov 03 21:30:00.739526 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.739474 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5"] Nov 03 21:30:00.743462 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.743440 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-xvscb_da9aeac9-c831-4be9-9aba-39a42918b4f6/istio-proxy/0.log" Nov 03 21:30:00.921377 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:00.921345 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt_e6cb547b-57ea-4b55-9d70-6597fa2073a7/istio-proxy/0.log" Nov 03 21:30:01.049201 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:01.049122 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" event={"ID":"97cf31cd-a03c-42b5-a04f-ff8b42b759e9","Type":"ContainerStarted","Data":"e5515c28e11b8c5e81a55633a57c06197181fd26b113478783200e28fec80347"} Nov 03 21:30:01.452018 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:01.451925 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-585db4bb4c-xs84v_837adf32-c68a-46ef-bca6-5b5364f74576/router/0.log" Nov 03 21:30:02.057480 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:02.057438 5815 generic.go:356] "Generic (PLEG): container finished" podID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerID="2436c1c1ce33fd044eac04ec82bfa7a8067bd97202ed650663042cb00c9764f8" exitCode=60 Nov 03 21:30:02.057939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:02.057529 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" event={"ID":"97cf31cd-a03c-42b5-a04f-ff8b42b759e9","Type":"ContainerDied","Data":"2436c1c1ce33fd044eac04ec82bfa7a8067bd97202ed650663042cb00c9764f8"} Nov 03 21:30:02.057939 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:02.057790 5815 scope.go:122] "RemoveContainer" containerID="2436c1c1ce33fd044eac04ec82bfa7a8067bd97202ed650663042cb00c9764f8" Nov 03 21:30:03.066434 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:03.066403 5815 generic.go:356] "Generic (PLEG): container finished" podID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerID="9f0a9c4a4e27e479209bf81631fd3edd75894a64308a10cd8178fd5fd341a147" exitCode=60 Nov 03 21:30:03.066922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:03.066488 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" event={"ID":"97cf31cd-a03c-42b5-a04f-ff8b42b759e9","Type":"ContainerDied","Data":"9f0a9c4a4e27e479209bf81631fd3edd75894a64308a10cd8178fd5fd341a147"} Nov 03 21:30:03.066922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:03.066576 5815 scope.go:122] "RemoveContainer" containerID="2436c1c1ce33fd044eac04ec82bfa7a8067bd97202ed650663042cb00c9764f8" Nov 03 21:30:03.066922 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:03.066741 5815 scope.go:122] "RemoveContainer" containerID="9f0a9c4a4e27e479209bf81631fd3edd75894a64308a10cd8178fd5fd341a147" Nov 03 21:30:03.067040 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:30:03.067000 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-jt9k5_odh-ai-gateway-infra(97cf31cd-a03c-42b5-a04f-ff8b42b759e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" Nov 03 21:30:04.073855 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:04.073830 5815 scope.go:122] "RemoveContainer" containerID="9f0a9c4a4e27e479209bf81631fd3edd75894a64308a10cd8178fd5fd341a147" Nov 03 21:30:04.074246 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:30:04.074032 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-jt9k5_odh-ai-gateway-infra(97cf31cd-a03c-42b5-a04f-ff8b42b759e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" Nov 03 21:30:08.024083 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:08.024049 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9"] Nov 03 21:30:08.026738 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:08.026714 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3400d899-278b9"] Nov 03 21:30:08.584695 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:08.584619 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42972838-3a85-4746-833e-f73b9d3f478e" path="/var/lib/kubelet/pods/42972838-3a85-4746-833e-f73b9d3f478e/volumes" Nov 03 21:30:13.851102 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:30:13.851066 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:30:17.067029 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.066993 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6k5bk_3b7ef6fe-524b-4542-b281-27f7fce2deb8/global-pull-secret-syncer/0.log" Nov 03 21:30:17.319217 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.319142 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6kg75_1551a4b4-7bd1-47ec-ae80-e79f4262761e/connection-checker/0.log" Nov 03 21:30:17.346074 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.346044 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7qnz8_148af517-1bbd-404e-97da-cadd491b1ded/connection-checker/0.log" Nov 03 21:30:17.373097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.373074 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fbn7r_844916e1-ce51-4692-907d-f30d1b6a9eb8/connection-checker/0.log" Nov 03 21:30:17.534328 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.534284 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cg74r_66b75f9f-c202-4a6c-b370-c5ce3d1de74c/konnectivity-agent/0.log" Nov 03 21:30:17.578425 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.578377 5815 scope.go:122] "RemoveContainer" containerID="9f0a9c4a4e27e479209bf81631fd3edd75894a64308a10cd8178fd5fd341a147" Nov 03 21:30:17.696730 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:17.696704 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-81.us-east-2.compute.internal_7974090a1f6621f0627618a748dcd304/haproxy/0.log" Nov 03 21:30:18.164439 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:18.164356 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" event={"ID":"97cf31cd-a03c-42b5-a04f-ff8b42b759e9","Type":"ContainerStarted","Data":"0ceab51b370a0209fc59870fb0de319b9d675e1e38242c3616b0f2ddcd7c84d8"} Nov 03 21:30:18.185129 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:18.185090 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" podStartSLOduration=18.185076995 podStartE2EDuration="18.185076995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:18.182499679 +0000 UTC m=+5327.712845348" watchObservedRunningTime="2026-11-03 21:30:18.185076995 +0000 UTC m=+5327.715422682" Nov 03 21:30:19.172696 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:19.172665 5815 generic.go:356] "Generic (PLEG): container finished" podID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerID="0ceab51b370a0209fc59870fb0de319b9d675e1e38242c3616b0f2ddcd7c84d8" exitCode=60 Nov 03 21:30:19.173110 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:19.172736 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" event={"ID":"97cf31cd-a03c-42b5-a04f-ff8b42b759e9","Type":"ContainerDied","Data":"0ceab51b370a0209fc59870fb0de319b9d675e1e38242c3616b0f2ddcd7c84d8"} Nov 03 21:30:19.173110 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:19.172795 5815 scope.go:122] "RemoveContainer" containerID="9f0a9c4a4e27e479209bf81631fd3edd75894a64308a10cd8178fd5fd341a147" Nov 03 21:30:19.173110 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:19.173039 5815 scope.go:122] "RemoveContainer" containerID="0ceab51b370a0209fc59870fb0de319b9d675e1e38242c3616b0f2ddcd7c84d8" Nov 03 21:30:19.173302 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:30:19.173281 5815 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895690-jt9k5_odh-ai-gateway-infra(97cf31cd-a03c-42b5-a04f-ff8b42b759e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" Nov 03 21:30:19.196787 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:19.196763 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5"] Nov 03 21:30:20.321658 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.321636 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:20.396591 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.396560 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca\") pod \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " Nov 03 21:30:20.397049 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.397023 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca" pod "97cf31cd-a03c-42b5-a04f-ff8b42b759e9" (UID: "97cf31cd-a03c-42b5-a04f-ff8b42b759e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:20.497304 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.497277 5815 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2\" (UniqueName: \"kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2\") pod \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\" (UID: \"97cf31cd-a03c-42b5-a04f-ff8b42b759e9\") " Nov 03 21:30:20.497577 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.497563 5815 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-maas-api-service-ca\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:20.499428 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.499395 5815 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2" pod "97cf31cd-a03c-42b5-a04f-ff8b42b759e9" (UID: "97cf31cd-a03c-42b5-a04f-ff8b42b759e9"). InnerVolumeSpecName "kube-api-access-rncv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:20.598046 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:20.598024 5815 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncv2\" (UniqueName: \"kubernetes.io/projected/97cf31cd-a03c-42b5-a04f-ff8b42b759e9-kube-api-access-rncv2\") on node \"ip-10-0-143-81.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:21.189066 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:21.189033 5815 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" Nov 03 21:30:21.189249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:21.189061 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5" event={"ID":"97cf31cd-a03c-42b5-a04f-ff8b42b759e9","Type":"ContainerDied","Data":"e5515c28e11b8c5e81a55633a57c06197181fd26b113478783200e28fec80347"} Nov 03 21:30:21.189249 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:21.189158 5815 scope.go:122] "RemoveContainer" containerID="0ceab51b370a0209fc59870fb0de319b9d675e1e38242c3616b0f2ddcd7c84d8" Nov 03 21:30:21.208312 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:21.208289 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5"] Nov 03 21:30:21.214851 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:21.214828 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895690-jt9k5"] Nov 03 21:30:22.584352 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:22.584318 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" path="/var/lib/kubelet/pods/97cf31cd-a03c-42b5-a04f-ff8b42b759e9/volumes" Nov 03 21:30:25.250801 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:25.250767 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-bx9l8_2276a55f-e1a9-406a-994e-ace6113b9d42/manager/0.log" Nov 03 21:30:25.280469 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:25.280442 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-jgwq4_df4b5345-2b3d-425d-a9f9-2dc3f382fcf7/manager/0.log" Nov 03 21:30:25.488458 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:25.488421 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-q9gdw_683dd503-5b16-4acc-9d0f-36fd1baf048e/limitador/0.log" Nov 03 21:30:25.639872 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:25.639791 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h946s_9da9ab34-02dc-458e-84d7-e9522d44c516/manager/0.log" Nov 03 21:30:27.439059 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:27.439027 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-txxdx_44590efe-c957-4995-8139-5b54567e51fd/cluster-monitoring-operator/0.log" Nov 03 21:30:27.806173 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:27.806149 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-smkqz_4783c1cb-2a65-4ab1-822a-e54f6d9ba570/node-exporter/0.log" Nov 03 21:30:27.834761 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:27.834738 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-smkqz_4783c1cb-2a65-4ab1-822a-e54f6d9ba570/kube-rbac-proxy/0.log" Nov 03 21:30:27.860205 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:27.860187 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-smkqz_4783c1cb-2a65-4ab1-822a-e54f6d9ba570/init-textfile/0.log" Nov 03 21:30:28.699191 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.699110 5815 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8"] Nov 03 21:30:28.699951 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.699927 5815 cpu_manager.go:414] "containerMap: removing container" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerName="cleanup" Nov 03 21:30:28.700062 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.699953 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerName="cleanup" Nov 03 21:30:28.700062 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.699971 5815 cpu_manager.go:414] "containerMap: removing container" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerName="cleanup" Nov 03 21:30:28.700062 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.699979 5815 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerName="cleanup" Nov 03 21:30:28.700231 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.700097 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerName="cleanup" Nov 03 21:30:28.700231 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.700113 5815 memory_manager.go:365] "RemoveStaleState removing state" podUID="97cf31cd-a03c-42b5-a04f-ff8b42b759e9" containerName="cleanup" Nov 03 21:30:28.704817 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.704797 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.711181 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.711157 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjmcb\"/\"openshift-service-ca.crt\"" Nov 03 21:30:28.711348 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.711332 5815 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjmcb\"/\"kube-root-ca.crt\"" Nov 03 21:30:28.711491 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.711469 5815 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fjmcb\"/\"default-dockercfg-sl6jl\"" Nov 03 21:30:28.714096 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.714073 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8"] Nov 03 21:30:28.766805 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.766776 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-podres\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.766957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.766811 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dtx\" (UniqueName: \"kubernetes.io/projected/59c11400-39b3-4657-a701-84db90f4287c-kube-api-access-n2dtx\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.766957 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.766848 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-lib-modules\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.767056 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.766977 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-proc\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.767056 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.767044 5815 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-sys\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.867861 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.867778 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dtx\" (UniqueName: \"kubernetes.io/projected/59c11400-39b3-4657-a701-84db90f4287c-kube-api-access-n2dtx\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.867861 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.867840 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-lib-modules\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.867881 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-proc\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.867952 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-sys\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.868005 5815 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-podres\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.868035 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-proc\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.868063 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-sys\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.868000 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-lib-modules\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.868097 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.868083 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59c11400-39b3-4657-a701-84db90f4287c-podres\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:28.879889 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:28.879868 5815 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dtx\" (UniqueName: \"kubernetes.io/projected/59c11400-39b3-4657-a701-84db90f4287c-kube-api-access-n2dtx\") pod \"perf-node-gather-daemonset-lnxt8\" (UID: \"59c11400-39b3-4657-a701-84db90f4287c\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:29.018832 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:29.018802 5815 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:29.296142 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:29.296110 5815 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8"] Nov 03 21:30:29.298149 ip-10-0-143-81 kubenswrapper[5815]: W1103 21:30:29.298122 5815 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod59c11400_39b3_4657_a701_84db90f4287c.slice/crio-66bed3bd42906ee2949eabe2e99a183d1596a134b7a64459b4283d8cde470756 WatchSource:0}: Error finding container 66bed3bd42906ee2949eabe2e99a183d1596a134b7a64459b4283d8cde470756: Status 404 returned error can't find the container with id 66bed3bd42906ee2949eabe2e99a183d1596a134b7a64459b4283d8cde470756 Nov 03 21:30:30.033260 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.033230 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-zwgdc_4faac008-58e3-41b1-9f56-8a8760071532/networking-console-plugin/0.log" Nov 03 21:30:30.247206 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.247172 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" event={"ID":"59c11400-39b3-4657-a701-84db90f4287c","Type":"ContainerStarted","Data":"498a1190a1e89702a6c6312e13c7eb2538681e20749b9b3c96572ffdb40d58ac"} Nov 03 21:30:30.247402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.247227 5815 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" event={"ID":"59c11400-39b3-4657-a701-84db90f4287c","Type":"ContainerStarted","Data":"66bed3bd42906ee2949eabe2e99a183d1596a134b7a64459b4283d8cde470756"} Nov 03 21:30:30.247402 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.247260 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:30.267192 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.267149 5815 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" podStartSLOduration=2.267136891 podStartE2EDuration="2.267136891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:30.264324147 +0000 UTC m=+5339.794669821" watchObservedRunningTime="2026-11-03 21:30:30.267136891 +0000 UTC m=+5339.797482564" Nov 03 21:30:30.871839 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.871814 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/2.log" Nov 03 21:30:30.879260 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:30.879239 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mxp5k_333ef57f-4887-4c8f-81c8-962265ad3be0/console-operator/3.log" Nov 03 21:30:31.023523 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:31.023484 5815 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8"] Nov 03 21:30:31.027944 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:31.027919 5815 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91e55dcb-764m8"] Nov 03 21:30:32.269252 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:32.269216 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-4gnrh_60d3aaa0-ccf0-42ae-92d5-44512e33aed4/volume-data-source-validator/0.log" Nov 03 21:30:32.583786 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:32.583713 5815 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df23b8b-3a84-48bc-aad8-fe9961216d50" path="/var/lib/kubelet/pods/4df23b8b-3a84-48bc-aad8-fe9961216d50/volumes" Nov 03 21:30:33.751981 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:33.751948 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6zdd_99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/dns/0.log" Nov 03 21:30:33.774079 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:33.774050 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6zdd_99e197f7-de69-4fb1-b6d8-e1ffaa6d4903/kube-rbac-proxy/0.log" Nov 03 21:30:33.898346 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:33.898318 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j4p9c_82de9774-f285-46a2-a1d0-cf66b6f780b2/dns-node-resolver/0.log" Nov 03 21:30:34.676463 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:34.676414 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n27x8_75f38eb4-b4e8-4673-a540-24c8ebe84ee8/node-ca/0.log" Nov 03 21:30:35.889309 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:35.889273 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-76f56c5945zkwwr_e5cb68d0-ef10-4bc6-9677-5eff1054ca0f/istio-proxy/0.log" Nov 03 21:30:36.105777 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:36.105711 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q5w7x_09dae28a-2531-444a-bf0c-bbe040b3b92d/manager/0.log" Nov 03 21:30:36.268632 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:36.268605 5815 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-lnxt8" Nov 03 21:30:40.525187 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:40.525155 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-76f56c5945zkwwr_e5cb68d0-ef10-4bc6-9677-5eff1054ca0f/istio-proxy/0.log" Nov 03 21:30:42.086421 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:42.086232 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q5w7x_09dae28a-2531-444a-bf0c-bbe040b3b92d/manager/0.log" Nov 03 21:30:42.768430 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:42.768391 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x96lq_b69e4a59-126c-4592-b7f5-b2ccf31781d7/discovery/0.log" Nov 03 21:30:43.472866 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:43.472823 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-gqpt8_ddfe5ba3-3ac6-4959-9ee5-6b15ce77aac9/istio-proxy/0.log" Nov 03 21:30:43.508272 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:43.508235 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-h8zwm_44953e3d-f118-443e-894d-530fba487260/istio-proxy/0.log" Nov 03 21:30:43.646093 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:43.646056 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-xvscb_da9aeac9-c831-4be9-9aba-39a42918b4f6/istio-proxy/0.log" Nov 03 21:30:43.676300 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:43.676266 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cd79c685f-znwwt_e6cb547b-57ea-4b55-9d70-6597fa2073a7/istio-proxy/0.log" Nov 03 21:30:43.784349 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:43.784231 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-585db4bb4c-xs84v_837adf32-c68a-46ef-bca6-5b5364f74576/router/0.log" Nov 03 21:30:43.857549 ip-10-0-143-81 kubenswrapper[5815]: E1103 21:30:43.857494 5815 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9p8gl" containerName="extractor" Nov 03 21:30:46.640865 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:46.640831 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mt748_0ed58f76-099a-498d-844d-0069904b507b/serve-healthcheck-canary/0.log" Nov 03 21:30:47.286010 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.285974 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-hft4t_f07a2feb-3648-4f19-9385-baeb5eeeaeeb/insights-operator/0.log" Nov 03 21:30:47.292887 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.292860 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-hft4t_f07a2feb-3648-4f19-9385-baeb5eeeaeeb/insights-operator/1.log" Nov 03 21:30:47.444067 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.444031 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9p8gl_17053989-3da4-4c70-92c3-949cd0b733ee/kube-rbac-proxy/0.log" Nov 03 21:30:47.472048 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.472014 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9p8gl_17053989-3da4-4c70-92c3-949cd0b733ee/exporter/0.log" Nov 03 21:30:47.506044 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.506009 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9p8gl_17053989-3da4-4c70-92c3-949cd0b733ee/extractor/0.log" Nov 03 21:30:47.654991 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.654911 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-cf8vr-g6gm2_25f7b1e8-8c25-4dbe-b4ff-e9c78d744b22/insights-gathering/0.log" Nov 03 21:30:47.800948 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:47.800918 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-q5w7x_09dae28a-2531-444a-bf0c-bbe040b3b92d/manager/0.log" Nov 03 21:30:48.039330 ip-10-0-143-81 kubenswrapper[5815]: I1103 21:30:48.039294 5815 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x96lq_b69e4a59-126c-4592-b7f5-b2ccf31781d7/discovery/0.log"